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SUMMARY 


The  efforts  to  integrate  artificial  intelligence  in  an  advanced  tactical 
fighter  (e.g.,  the  Pilot’s  Associate)  are  likely  to  be  hampered  by  two 
potentially  serious,  but  avoidable  shortcomings.  The  first  involves  a  failure 
to  adequately  incorporate  multiple  pilot  perspectives  in  the  design  of  the 
system,  and  the  second  involves  the  insufficient  breadth  of  the  knowledge¬ 
base.  Without  the  inclusion  of  multiple  pilot  perspectives,  there  is  a 
significantly  greater  risk  that  the  system  will  fail  to  address  the  pilot’s 
information  requirements  for  a  tactical  fighter  mission,  and  thereby  fail  to 
satisfy  the  pilot’s  requirement  tor  a  Pilot’s  Associate.  Knowledge-based 
systems  which  fail  to  derive  common-sense  world  knowledge,  analogies, 
heuristics,  beliefs,  and  the  experience  which  underlies  an  expert’s 
performance,  run  the  significant  risk  of  displaying  brittleness  when 
confronted  with  real  world  performance  settings.  These  issues  may  be 
addressed  by:  1)  providing  the  PA  knowledge  engineers  with  the  knowledge 
elicitation  tools  necessary  to  acquire  and  incorporate  multiple  pilot 
perspectives,  and  2)  developing  a  large  knowledge-base  whose  breadth  and 
depth  are  sufficient  to  handle  the  complexity  associated  with  the  tactical 
fighter  mission. 

The  research  described  in  this  paper  is  intended  to  address  these 
shortcomings  through  the  development  and  utilization  of  an  innovative 
knowledge  and  design  acquisition  methodology  that  is  intended  to:  1) 
highlight  the  domain  expert’s  conceptualization  of  the  problem  domain;  2) 
identify  the  information  requirements;  3)  elicit  from  the  domain  expert 
design  prototypes  and  evolve  these  design  prototypes  using  the  design 
storyboarding  technique;  and  4)  document  the  rationale  behind  the 
information  requirements  and  design. 

The  knowledge  and  design  acquisition  methodology  is  comprised  of 
three  components,  the  concept  mapping  technique,  IDEF0  modeling 
technique  and  design  storyboarding.  This  paper  is  chiefly  a  concept 
demonstration  effort  which  explores  the  prospective  utility  of  these 

techniques.  j 
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INTRODUCTION 


1.0  Background 

One  of  the  major  research  and  development  projects  in  applied 
artificial  intelligence  today  is  the  DARPA  sponsored  Pilot’s  Associate  (PA) 
program.  For  several  years,  scientists  at  the  Wright  Research  and 
Development  Center  (WRDC)  have  been  involved  in  the  design  of  cooperative 
knowledge-based  systems  which  would  culminate  in  a  sophisticated 
electronic  crew  member  to  interact  with  the  pilot  during  a  single  seat, 
tactical  fighter  mission.  The  intent  underlying  the  PA  program  as  outlined 
by  Lizza  (1989)  revolves  around  the  provision  of  a  technology  pull  for  the 
Strategic  Computing  program  to  explore  the  potential  of  artificial 
intelligence  to  improve  mission  effectiveness  and  survivability  of  advanced 
fighter  aircraft.  Within  this  exploration,  there  are  many  technological  and 
programmatic  issues  which  may  act  as  barriers  to  the  eventual  flight  test  of 
such  a  system  (scheduled  for  the  mid  1990s).  One  of  the  most  important 
areas  of  concern  is  the  myriad  of  relationships  which  may  develop  between 
the  pilot  and  the  associate  and  the  ways  in  which  these  relationships  are 
manifest  through  an  interface  (i.e.,  the  Pilot- Vehicle  Interface).1 

The  psychology  of  the  interaction  between  the  pilot  and  the  associate  is 
a  direct  product  of  the  design  of  the  PA.  Comparatively  little  attention  has 
been  given  to  this  interaction,  and  as  a  consequence,  there  is  a  distinct 
possibility  that  the  design  of  the  PA  may  transpire  without  recognition  of 
the  importance  of  the  pilot’s  role  in  the  interaction.  Before  discussing  the 
pilot’s  requirements,  and  how  they  should  impact  the  design  process,  it  is 
necessary  to  describe  what  is  meant  by  a  PA . 


'  Several  papers  have  taken  directions  with  respect  to  such  issues.  For  example,  Rouse 
(1988)  addresses  the  adaptive  aiding  aspects  of  human/computer  control,  McNeese  (1986) 
addresses  the  ‘combined  intelligence’  of  pilots  and  associates,  Snyder  &  McNeese  (1987) 
analyze  conflicts  in  cooperative  man-machine  systems,  Snyder,  Brown,  Wellens,  and 
McNeese  (1989)  discusses  the  distributed  decision  making  paradigms  for  studying 
human-to-human  and  human-to-intelligent  machine  relationships,  and  Wellens  & 
McNeese  (1987)  review  the  social  psychology  of  integrating  intelligent  associates  with 
humans;  to  cite  some  of  the  different  directions  which  have  evolved  from  the  seminal  PA 
program. 
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The  PA  is  being  designed  as  a  collection  of  coupled  expert  systems  to 
provide  real-time  assistance  to  a  pilot  of  advanced  single-seat  fighters.  The 
PA  will  organize,  filter,  integrate,  and  prioritize  data,  and  then,  provide  the 
pilot  with  essential  information,  assistance  and  advice.  It  will  take 
advantage  of  artificial  intelligence  technology  with  the  intention  of  using  a 
knowledge  base  that  contains  the  types  of  knowledge  needed  to  enhance  the 
fighter  pilot's  performance.  It  will  make  use  of  the  symbolic  and  logical 
processing  capabilities  particular  to  expert  systems  to  assist  in  the 
performance  of  many  data  analysis  and  logical  decision  making  tasks 
presently  accomplished  by  the  pilot,  as  well  as  some  that  are  currently 
beyond  his  or  her  capabilities.  The  pilot  will  be  able  to  call  upon  the 
associate  for  certain  function  allocation  tradeoffs  as  needed  in  situations  of 
high  stress  and  workload,  and  during  times  of  impending  lack  of  situation 
awareness. 

Small,  Lizza,  &  Zenyuh  (1989)  indicate  that  the  PA  contains  six 
cooperating  expert  systems  for  pilot  decision  support  which  include:  the 
Mission  Planner,  the  Tactics  Planner,  Situation  Assessment,  Systems 
Status,  Pilot- Vehicle  Interface,  and  Mission  Executive.  These  systems 
exchange  information  as  needed  and  are  propagated  with  changing 
mission  data  and  the  environmental  context  to  support  the  pilot.  Figure  1-1 
shows  the  overall  concept  of  a  PA,  including  the  proposed  subsystems. 

The  article  by  Small,  Lizza  and  Zenyuh  (1989)  provides  an  in-depth 
description  of  the  proposed  functionality  of  each  of  these  subsystems,  and 
should  be  consulted  for  further  information  regarding  characteristics  of 
these  subsystems.  A  brief  description  of  each  subsystem  based  on  the 
review  by  Lizza  &  Friedlander  (1988)  is  shown  in  Table  1-1. 


1.1  Statement  of  the  problem 

In  addition  to  the  technical  obstacles  facing  the  PA,  the  efforts  to 
integrate  artificial  intelligence  in  an  advanced  tactical  fighter  are  likely  to 
be  hampered  by  two  other  potentially  serious,  but  avoidable  shortcomings. 
The  first  involves  a  failure  to  adequately  incorporate  multiple  pilot 
perspectives  in  the  design  of  the  system,  and  the  second  involves  the 
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Figure  1-1  PA  Architecture 


Table  1-1.  PA  Subsystems 


Systems  status  monitors  on-board  components  to  identify, 
diagnose,  and  verify  system  malfunction  and  to  determine  the 
appropriate  compensation  mechanism. 

Situation  Assessment  provides  an  accurate  and  coherent 
assessment  of  the  type,  position,  and  intent  of  external  entities 
effecting  the  planned  mission  wherein  the  understanding  of 
threats  and  prediction  of  future  actions  may  occur. 

Mission  Planner  compares  preplanned  mission  models  with 
actual  events  to  evaluate  the  impact  of  new  data  and  provides  a 
modification  of  the  mission  plan  as  appropriate. 

Tactics  Planner  provides  the  pilot  with  specific  information 
and  actions  (e.g.,  suggestions  of  maneuvers,  weapons  and 
countermeasure  employment,  and  sensor  use)  regarding 
threats  and  targets  and  coordinates  tactics  for  multi-aircraft 
flights. 

Pilot- Vehicle  Interface  is  the  communications  medium 
between  the  pilot  and  the  associate  in  which  intelligent 
computing  functions  adaptively  control  the  display  content  and 
timing,  information  flow,  and  allow  the  pilot  to  control  the 
associate  dependent  on  assessment  or  estimation  of  workload, 
cognitive  resources,  pilot  preferences,  and  inferred  pilot 
intent. 

Mission  Executive  is  responsible  for  ensuring  the  smooth 
operation  and  control  of  the  PA  through  the  use  of  high-level 
conflict  resolution,  resource  allocation,  global  strategy 
development,  and  task  scheduling  for  real  time  operations;  it 
also  maintains  the  mission  blackboard  as  the  message  center 
of  the  PA.  Taken  together,  these  cooperating  expert  systems 
provide  the  functions  and  structures  for  the  PA  architecture. 


Note  that  the  PA  is  being  designed  to  be  included  as  a  real 
time,  on-board  associate.  Taken  together,  these  cooperating 
expert  systems  provide  the  functions  and  structures  for  the  PA 
architecture. 
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creation  of  coupled  expert  systems  with  insufficient  breadth  and  depth  of 
the  resultant  knowledge  base. 

Given  the  tremendous  complexity  involved  in  the  design  and 
implementation  of  the  PA’s  architecture,  it  is  easy  to  become  preoccupied 
with  attempts  to  solve  the  technical  issues,  and  lose  sight  of  the  pilot’s 
perspective  on  the  system  requirements.  Lizza  (1989),  in  a  review  of  the 
current  status  of  the  PA  program  identified  knowledge  acquisition  as  an 
area  of  major  concern,  with  his  recognition  that  the  existing  approach  was 
running  the  risk  of  failing  to  adequately  incorporate  the  pilot’s  perspective. 
He  suggested  that  although  knowledge  acquisition  needs  to  be  well 
disciplined  and  carefully  structured,  the  prior  efforts  which  have  adopted  a 
top-down  knowledge  acquisition  approach,  were  basically  flawed  to  the 
extent  that  the  elicitation  process  has  produced  knowledge  which  cannot  be 
implemented  using  classical  rule-based  or  frame-based  approaches  to 
expert  system  design.  Lizza  goes  on  to  suggest  that  what  is  needed  is  a 
bottom-up,  scenario- driven  approach  to  knowledge  acquisition  that 
establishes  a  context  and  framework  for  eliciting  knowledge  through 
interviews.  Recognition  of  the  deficiencies  regarding  methodologies  used 
for  knowledge  acquisition  served  as  the  original  impetus  for  our  research 
efforts.  These  efforts  are  directed  toward  the  development  of  an  integrated 
knowledge  acquisition  methodology  and  the  establishment  of  an  advanced 
user  requirements  analysis  and  synthesis  framework  for  the  Pilot- Vehicle 
Interface  (PVI). 

The  second  shortcoming  currently  facing  the  Pilot’s  Associate 
program  involves  the  creation  of  coupled  expert  systems  whose  architecture 
consists  of  a  collection  of  narrow,  task-specific  problem  solvers.  As  a 
consequence,  the  expert  systems  will  generally  attempt  to  finesse 
intelligence  in  areas  where  knowledge:  1)  is  weak  and  ill-defined,  2)  is  not 
elicited  in  depth  from  the  domain  experts,  3)  is  captured  without  sufficient 
inter-connectivity  or  contextual  indexing,  and  4)  is  under-represented  (i.e., 
large  cases  of  understanding  are  not  developed).  Lizza  &  Friedlander  (1988) 
concluded  that  the  next  major  hurdle  for  the  contractors  implementing  the 
PA  is  the  acquisition  of  large  volumes  of  knowledge  from  experts.  It  is  our 
belief  that  the  development  of  the  PA  must  be  based  upon  the  pilot’s  ability 
to:  1)  match  situations  and  adjust  slightly  (remembrance),  2)  match  far 
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flung  situations  (analogize),  3)  fall  back  on  general  knowledge  (use 
common  sense),  and  4)  learn  more  about  situations  (recursion);  (adapted 
from  Lenat  &  Guha,  1990).  The  attributes  to  supplement  these  abilities 
must  be  designed  into  the  PA  system  from  a  pilot’s  perspective.  Knowledge- 
based  systems  must  utilize  common-sense  world  knowledge,  analogies, 
heuristics,  beliefs,  and  the  experience  which  underlies  an  expert’s 
performance,  or  they  run  a  significant  risk  of  displaying  brittleness  when 
confronted  with  real  world  performance  settings. 

Our  efforts  to  develop  a  new  knowledge  acquisition  methodology  have 
focused  on  the  premise  that  a  development  project  such  as  the  PA  Program 
must  proceed  from  the  pilot’s  conceptualization  of  his  or  her  mission.  This 
conceptualization  must  direct  the  development  of  the  PA,  and  the  thoughts, 
ideas,  or  hunches  (i.e.,  intuitive  knowledge),  as  well  as  the  experiential 
knowledge  of  the  pilot  must  be  explicitly  identified  and  shared  with  the 
other  members  of  the  design  team.  Without  the  inclusion  of  multiple  pilot 
perspectives  within  the  PA  design,  there  is  a  significant  risk  that  the 
system  will  fail  to  address  the  pilot’s  information  requirements  for  a 
tactical  fighter  mission,  and  thereby  fail  to  satisfy  the  pilot’s  requirement 
for  a  Pilot’s  Associate. 

In  addressing  the  shortcomings  facing  the  PA  program,  we  have 
pursued  the  development  and  evaluation  of  knowledge  acquisition  tools  that 
are  designed  to  capture  the  pilot’s  comprehension  of  the  tactical  fighter 
mission.  Our  intention  is  to  incorporate  multiple  knowledge  representation 
techniques,  as  well  as  automating  portions  of  the  knowledge  acquisition 
process  in  order  to  facilitate  the  rapid  collection,  organization,  and  analysis 
of  the  inputs  from  numerous  pilots. 


1.2  Uncovering  the  Expert’s  Knowledge 

The  efforts  outlined  in  this  report  are  directed  at  providing  a 
methodology  that  can  be  used  to  identify  the  user’s  requirements  for  the 
PA/Pilot- Vehicle  Interface.  This  methodology  is  intended  to  be  useful  for 
uncovering  the  pilot’s  understanding  of  the  mission  by  focusing  on  the 
pilot’s  intuitive  and  experiential  knowledge  which  often  fails  to  appear  in 
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more  structured  forms  of  knowledge  representation.  To  the  extent  that  this 
methodology  has  been  applied  during  this  initial  concept  demonstration 
phase,  it  will  enable  the  developers  of  a  Pilot’s  Associate  to  satisfactorily 
answer  the  following  questions:  1)  What  information  does  the  pilot  expect  to 
get,  when,  in  what  form,  and  by  whom?  2)  When  and  what  must  the  pilot 
anticipate  doing  with  the  information  in  order  to  accomplish  the  mission 
goals?  3)  How  should  the  function  allocation  vary  given  changing 
situations?  4)  How  should  the  pilot  communicate  with  the  associate? 
McCormick  (1964)  restates  these  issues  as  he  suggests  that  “in  terms  of 
design  considerations,  it  is  necessary  to  anticipate  who  (or  what)  is  to  ‘talk’ 
to  whom  (or  what)  and  to  provide  for  an  appropriate  link  to  make  this 
happen”  (p.  10). 

The  design  of  the  pilot’s  associate  has  not  proceeded  without  various 
attempts  to  capture  the  pilot’s  knowledge  regarding  the  fighter  mission. 
Unfortunately  these  efforts  have  generally  relied  upon  a  top-down 
knowledge  acquisition  approach  (i.e.,  mission  or  task  decomposition)  and 
resulted  in  a  highly  structured  knowledge  representation.  Often  these 
attempts  are  taken  from  the  engineer’s  or  analyst’s  viewpoints,  and 
although  deriving  specific  mission  requirements,  they  fail  to  adequately 
represent  the  pilot’s  own  conceptualization  of  his  mission.  In  fact,  some  of 
the  potential  problems  currently  facing  the  PA  development,  as  identified  by 
Lizza  (1989),  may  be  a  direct  consequence  of  basing  the  design  of  this  AI 
system  on  the  engineer’s  and  analyst’s  understanding  of  the  vehicle, 
environment  and  mission  rather  than  upon  the  intended  user  of  this 
domain.  It  is  our  belief  that  the  PA/Pilot- Vehicle  Interface  must 
successfully  meet  the  user’s  requirements,  and  must  therefore  be  based  on 
the  user’s  understanding  of  vehicle,  environment,  mission,  and  associate. 

In  an  effort  to  deal  with  the  shortcomings  in  the  area  of  knowledge 
acquisition,  and  to  adequately  elicit  the  user’s  perspective  of  the  problem 
domain,  an  approach  was  developed  to  extract  both  general  and  specific 
knowledge  from  pilots  for  a  mission  segment  appropriate  to  the  PA 
program  objectives.  The  intent  was  to  continually  evolve  the  PA  system 
design  requirements  on  the  basis  of  this  acquired  knowledge.  Specifically, 
the  efforts  described  in  this  report  involve:  1)  the  identification  of  the  key 
decision  points  and  information  requirements  from  the  pilot’s  perspective, 
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and  2)  the  construction  of  Pilot- Vehicle  Interface  design  storyboards.  Both 
of  these  efforts  involved  the  use,  and  subsequent  integration,  of  several 
“handcrafted”  knowledge  acquisition  techniques.  The  projected  payoffs 
from  the  application  of  these  knowledge  acquisition  techniques  in  an 
integrated  methodology  include:  1)  an  expanded  identification  of 
information  required  for  the  pilot  to  make  decisions/actions;  2)  specific 
knowledge  pertaining  to  situation  awareness  (i.e.,  where  the  pilot  is 
focusing  his  attention  during  various  points  in  the  mission);  3)  a  basis  for 
function  allocation  between  the  pilot  and  the  associate;  and  4)  a  framework 
for  designing  and  evaluating  interface  prototypes.  These  efforts  will 
eventually  culminate  with  the  creation  of  an  automated  knowledge 
acquisition  system  which  will  be  capable  of  generating  an  in-depth 
knowledge  base  to  counteract  some  of  the  inadequacies  previously 
associated  with  brittle  expert  systems. 

In  addition  to  meeting  the  PA  program’s  need  for  an  innovative 
knowledge  acquisition  methodology,  our  research  efforts  are  intended  as  a 
direct  response  to  the  recently  published  DOD  Critical  Technologies  Plan 
(1990).  This  plan  was  provided  to  congress  as  part  of  an  overall  Science  and 
Technology  investment  strategy  derived  from  the  National  Military 
Strategy,  published  by  the  Joint  Chiefs  of  Staff,  the  Defense  Planning 
Guidance,  and  the  Office  of  the  Secretary  of  Defense. 

The  first  critical  technology  our  research  effort  supports  is  Critical 
Technology  No.4,  Machine  Intelligence  and  Robotics  which  lists  knowledge 
acquisition,  knowledge  representation,  automated  reasoning,  improved 
man-machine  interfaces,  and  training  as  the  major  challenges  facing  this 
area.  The  issues  that  our  research  address  are  heavily  entwined  with  such 
challenges.  The  second  critical  technology  which  our  research  supports  is 
Critical  Technology  No.  5,  Simulation  and  Modeling.  This  plan  encourages 
the  application  of  Artificial  Intelligence  and  object-oriented  programming 
to  create  easy  and  affordable  simulations  and  computer-based  models  that 
mimic  the  behavior  of  real  objects.  The  plan  references  the  potential  payoff 
of  simulation  and  modeling,  including  the  behavioral  modeling  of  crew 
performance  and  the  development  of  computer-aided  decision  support 
systems  as  a  means  of  addressing  human  factors  issues  in  the  combat 
environment.  Additionally,  the  plan  emphasizes:  1)  the  role  of  virtual 
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prototyping  as  an  effective  method  of  visualizing  system  components  in  an 
effort  to  reduce  costs  and  deliver  the  final  product  in  a  quicker  time  frame; 
2)  new  weapon  systems  design  (including  human  factors  and  training 
considerations  via  computer  simulations)  for  the  purpose  of  determining 
design  effectiveness,  learning  constraints,  and  cognitive  overload 
considerations;  and  3)  the  creation  of  new  design  practices  to  permit 
experimentation  prior  to  full-scale  development  of  a  new  system. 

Our  use  of  integrated  knowledge  acquisition  and  representation 
techniques,  as  well  as  storyboard  prototyping,  within  an  object-oriented 
design  environment  meets  the  vision  described  in  this  DOD  plan.  Our 
agenda  for  future  developments  in  the  areas  of  automated  knowledge  and 
design  acquisition  technology  are  directly  integrated  with  the  critical 
technologies  mentioned. 


1.3  Knowledge  Acquisition  Techniques 

Knowledge  acquisition  techniques  take  many  different  forms.  In  fact, 
classifying  and  categorizing  the  myriad  of  techniques  and  tools  has  been 
the  focus  of  attention  for  several  investigators  (Boose,  1989;  Hoffman,  1987; 
Bloomfield  and  Shalin,  1989;  Mitta,  1989;  Boehm-Davis,  1989).  As  it  is  not 
possible  to  discuss  all  the  available  tools  and  methods  that  have  been  used 
or  developed  to  elicit  expert  knowledge,  we  will  limit  our  discussion  to  the 
specific  direct  and  indirect  methodologies  that  could  be  used  in  the  present 
context.  That  is,  techniques  that  rely  on  direct  interaction  with  the 
knowledge  engineer,  those  that  rely  on  introspection  on  the  part  of  the 
domain  expert,  and  those  that  emphasize  the  domain  characteristics  of 
naturalistic  decision-making  in  dynamic  environments  will  be  of 
particular  interest. 

1.3.1  Direct  Methods 

Direct  methods  of  knowledge  acquisition  include  techniques  in  which 
experts  are  required  to  report  experiences  in  using  a  system  or 
accomplishing  a  particular  task.  Interviews  (structured  or  unstructured), 
verbal  protocols,  and  questionnaires  fall  into  this  category.  In  a  generic 
sense,  these  direct  knowledge  acquisition  strategies  are  the  techniques  of 
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primary  interest  because  of  their  frequent  use  in  domains  that  are 
characterized  by  natural  decision  making  in  a  dynamic  environment. 

Structured  interviews  involve  asking  experts  specific  questions  about 
the  problem  domain  of  interest.  Questions  may  take  the  form  of  rules, 
procedures,  object  relationships,  etc.  In  this  case,  the  knowledge,  engineer 
must  first  construct  these  questions  from  his/her  own  knowledge  base.  The 
information  may  come  from  system  documentation,  rules  of  engagement, 
or  procedural  documentation.  For  a  structured  interview,  the  knowledge 
engineer  must  have  a  fairly  complete  understanding  of  the  domain  in  order 
to  develop  the  necessary  set  of  queries  to  acquire  the  expert’s  knowledge.  It 
is  generally  agreed  that  the  quality  of  the  data  collected  is  highly  dependent 
upon  the  amount  of  domain  knowledge  that  the  knowledge  engineer 
possesses  The  domain  expert's  task  is  limited  to  the  confirmation  and/or 
correction  of  information  in  response  to  direct  questions. 

Unstructured  interviews  have  an  open-ended  format.  The  knowledge 
engineer  probes  specific  areas  of  interest  in  response  to  the  domain  expert's 
description  of  the  task  or  system.  Presumably,  the  knowledge  engineer  also 
has,  at  least,  a  working  knowledge  of  the  domain  in  question  in  order  to 
probe  particular  areas  of  interest  and  to  allow  for  interpretation  of  the 
information  the  domain  expert  is  producing.  In  addition  to  requiring  a 
considerable  degree  of  sophistication  on  the  part  of  the  interviewer 
developing  questions,  both  structured  and  unstructured  interview 
techniques,  because  of  their  reliance  upon  the  questions  that  the  knowledge 
engineer  generates,  run  the  significant  risk  of  biasing  the  knowledge 
acquisition  process. 

Verbal  protocol  techniques  are  examples  of  unstructured  interviews 
which  attempt  to  avoid  the  biases  that  are  generated  by  the  knowledge 
engineer’s  particular  line  of  questioning.  In  these  techniques,  the  domain 
expert  is  encouraged  to  "talk  through"  a  task  or  procedure,  and  to  describe 
what  is  ‘going  through  his  or  her  mind’  as  he  or  she  is  performing  the 
task.  The  elicitation  of  expert  knowledge  by  verbal  protocol  consists  of  the 
domain  expert  verbally  articulating  the  sequence  of  events  of  a  task  with 
which  he  or  she  is  engaged.  With  the  use  of  a  verbal  protocol  technique, 
some  of  the  decision  processes  that  are  occurring  as  the  task  is  being 


10 


accomplished  are  captured  in  the  domain  expert’s  verbalizations.  These 
verbalizations  are  recorded  for  detailed  analysis  at  a  later  date. 

There  are  a  number  of  verbal  protocol  techniques,  including  Method  of 
Familiar  Tasks,  Limited  Information  Tasks,  Constrained  Process  Tasks, 
and  Method  of  Tough  Cases  (see  Hoffman  1987),  that  have  a  common 
characteristic.  The  domain  expert  is  asked  to  report  directly  on  his  or  her 
experiences  in  accomplishing  a  specific  task.  The  Method  of  Familiar 
Tasks  involves  an  analysis  of  the  tasks  that  the  expert  usually  performs. 
What  the  expert  knows  is  typically  represented  in  terms  of  prepositional 
statements  that  are  meaningfully  related  to  the  domain  in  question.  The 
Limited-Information  Task  involves  a  variation  of  the  Familiar  Task 
approach,  in  which  the  domain  expert  is  asked  to  perform  a  familiar  task 
that  has  been  manipulated  so  as  to  limit  the  available  data.  The 
assumption  regarding  this  manipulation  is  that  with  a  limited  amount  of 
information  the  domain  expert  will  be  forced  to  decompile  and  reason  about 
what  would  otherwise  be  a  highly  automated  mental  process.  Constrained 
Processing  tasks  also  utilize  and  manipulate  familiar  tasks.  However,  the 
manipulation  typically  involves  the  imposition  of  time  constraints  or 
resource  limitations.  Method  of  Tough  Cases  is  yet  another  variation  of 
verbal  protocol  technique.  In  this  case,  however,  the  expert  verbalizes 
procedures  while  accomplishing  a  non-routine  or  unusual  occurrence  of  a 
familiar  task. 

While  the  verbal  protocol  techniques  may  successfully  avoid  the 
introduction  of  the  knowledge  engineer’s  biases  as  a  consequence  of  the 
questions  that  are  being  asked,  they  do  have  several  other  inherent 
shortcomings.  First,  these  techniques  often  are  used  to  elicit  knowledge 
that  the  domain  expert  is  likely  to  have  difficulty  articulating  due  to  the  fact 
that  much  of  the  knowledge  is  tacit  in  nature  (Polany,  1966).  Tacit 
knowledge  is  learned  by  watching  or  doing,  as  opposed  to  being  taught  by 
verbal  instruction.  Second,  and  perhaps  more  serious,  is  the  fact  that  the 
obtrusiveness  of  these  techniques  may  have  a  major  impact  on  the  task 
strategies,  by  altering  the  typical  expression  of  the  expert’s  behavior. 
Additional  problems  may  occur  involving  the  process  of  interpreting  the 
verbalizations.  The  process  is  extremely  time  consuming,  and  runs  the 
unfortunate  risk  of  re-introducing  knowledge  engineer  biases  due  to  the 
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fact  that  the  knowledge  engineer  is  prone  to  interpret  the  domain  expert’s 
comments  from  her/his  own  frame-of-reference. 

A  variation  of  the  Method  of  Tough  Cases  is  the  Critical  Decision 
Method  (CDM)  which  uses  a  set  of  cognitive  probes  to  determine  the  basis 
for  situation  assessment  and  decision  making  during  non-routine 
incidents  (Klein,  Calderwood,  and  MacGregor,  1989).  In  contrast  to  the 
verbal  protocol  techniques  in  which  the  experts  try  to  articulate  the  task  and 
decision  processes  as  they  occur,  the  CDM  relies  on  interviews  with  domain 
experts  that  examine  recent  cases  of  interest.  The  recent  cases  of  interest 
are  non-routine  incidents,  rather  than  tasks  that  occur  regularly.  This 
probing  of  specific  incidents  is  said  to  result  in  data  that  are  superior  in 
kind  and  quality  to  that  gained  eliciting  knowledge  about  general  rules  or 
procedures  (Klein  et.  al.,1989).  While  the  CDM  may  provide  knowledge  of 
appreciable  depth,  the  focus  on  non-routine  decisions  that  occur  in  the 
course  of  dynamic  situations,  runs  the  risk  of  failing  to  acquire  a  sufficient 
breadth  of  knowledge.  Like  the  various  verbal  protocol  techniques,  the 
expert  would  be  limited  to  a  single  occurrence  of  a  task  and,  therefore,  the 
efficiency  of  the  knowledge  acquisition  efforts  would  be  reduced. 

1.3.2  Indirect  Methods 

Indirect  techniques  include  observational  studies,  simulations  and 
other  unobtrusive  methods  that  note  and  analyze  response  patterns. 

Number  of  responses,  errors,  and  latency  of  response  are  the  type  of  data 
collected  in  these  cases.  These  types  of  techniques  do  not  rely  on 
verbalization  or  discussion  with  the  knowledge  engineer  during  task  action 
or  decision  making.  Therefore,  there  is  less  likely  a  chance  that 
interference  will  affect  the  strategies  or  reasoning  processes  of  the  domain 
expert.  However,  only  inferences  about  tacit  knowledge,  strategies,  and 
processes  can  be  made  with  indirect  techniques.  Therefore,  missing  or 
misleading  expert  knowledge  may  result. 


1.4  Requirements  for  an  Integrated  Knowledge  Acquisition  Methodology 

A  commitment  to  elicit  knowledge  from  the  domain  expert  and  to 
codify  that  knowledge  in  an  intelligent  system  does  not  however,  in  and  of 
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itself,  insure  that  the  domain  expert’s  perspective  and  understanding  of  the 
problem  domain  will  be  faithfully  represented.  Highly  structured 
techniques  and  those  relying  principally  upon  informed  access  run  the 
significant  risk  of  biasing  the  domain  expert’s  presentation  and  recall  of 
knowledge.  Informed  access  refers  to  situations  in  which  the  domain 
expert’s  knowledge  is  elicited  as  a  function  of  the  knowledge  engineer’ s 
probes  rather  than  being  spontaneously  accessed  by  the  expert  (i.e., 
uninformed  access).2 

A  knowledge  acquisition  process  that  begins  with  informed  access  is 
likely  to  elicit  information  from  the  domain  expert  that  is  in  concordance 
with  the  knowledge  engineer’s  prior  conceptualization  of  the  problem 
domain.  Since  the  knowledge  engineer  directs  the  knowledge  acquisition 
process  by  informing  the  domain  expert  of  the  information  desired,  the 
information  elicited  from  the  domain  expert  is  likely  to  be  a  reflection  of 
what  the  knowledge  engineer  considers  to  be  important,  rather  than  a 
faithful  representation  of  the  relevant  information  and  predominant  issues 
from  the  domain  expert’s  perspective.  When  the  knowledge  acquisition 
process  begins  with,  or  relies  exclusively  upon  techniques  that  involve 
either  structured  or  unstructured  questioning  of  the  domain  expert,  the 
risk  of  biasing  the  outcome  is  significant.  Informed  access  creates  a 
knowledge  acquisition  setting  which  may  be  artificially  constrained  in  that 
it  fails  to  capture  the  expert’s  own  necessity  to  recall  knowledge  as  needed 
without  being  informed  to  do  so  (as  is  often  the  case  in  real  world  settings). 

In  contrast,  uninformed  access  involves  the  unprompted  elicitation  of 
the  concepts  that  are  used  by  the  domain  expert,  and  which  arise  from  his 
or  her  own  unbiased  perspective  of  the  concepts  associated  with  the  problem 
domain.  Uninformed  access  allows  the  presentation  of  pilot  knowledge  to 
arise  directly  from  his  experience  with  the  problem  domain.  The 
distinction  between  informed  and  uninformed  access  can  be  understood  in 
relation  to  a  courtroom  proceeding  in  which  the  informed  access  is 
analogous  to  a  situation  where  the  trial  lawyer  is  leading  the  witness,  and 
uninformed  access  is  analogous  to  the  presentation  and  recall  of  an 
unbiased  eyewitness  testimony.  In  a  courtroom  setting  there  would 

*  See  Perfetto,  Bransford,  &  Franks,  1983;  Adams,  Kasserman,  Yearwood,  Perfetto, 
Bransford,  &  Franks,  1988  for  research  relating  to  constraints  on  knowledge  acquisition 
and  access. 


13 


undoubtedly  be  a  difference  between  the  veridicality  of  the  two  testimonies, 
and  likewise,  in  the  field  of  knowledge  acquisition  there  is  a  difference 
between  the  degree  to  which  the  knowledge  engineer  biases  the  domain 
expert’s  presentation  when  using  the  two  different  approaches. 

The  intent  of  distinguishing  between  informed  and  uninformed 
access,  is  to  point  out  the  risks  associated  with  the  usage  of  informed  access 
interview  techniques  as  the  sole  method  of  knowledge  acquisition.  To  elicit 
an  unbiased  representation  of  the  expert’s  understanding  of  the  problem 
domain,  the  knowledge  acquisition  procedure  should  begin  by  first  eliciting 
the  concepts  that  are  spontaneously  accessed  by  the  domain  expert  under 
uninformed  conditions. 

There  is  a  distinct  value  gained  by  probing  the  domain  expert  for 
details  and  greater  depth  of  knowledge,  and  the  knowledge  acquisition 
process  cannot  be  confined  to  a  method  that  employs  only  an  uninformed 
access  to  domain  knowledge.  Rather,  a  method  must  be  employed  that 
utilizes  both  informed  and  uninformed  access.  The  knowledge  acquisition 
process  should  begin  with  an  uninformed  access,  to  insure  an  unbiased 
presentation  of  the  concepts  that  the  domain  expert  considers  to  be 
important.  As  the  elicitation  process  continues  there  will  be  more 
opportunity  for  direction  and  structure  to  be  provided  by  the  knowledge 
engineers  (e.g.,  specific  probes)  thereby  creating  a  setting  of  informed 
access.  Once  the  domain  expert  has  had  the  opportunity  to  thoroughly 
discuss  his  or  her  understanding  of  the  problem  domain,  it  becomes  in 
essence  ‘safe’  for  the  knowledge  engineer  to  begin  probing  for  additional 
information  without  a  significant  risk  of  biasing  the  concepts  that  are 
presented. 


1.5  An  Integrated  Knowledge  Acquisition  Framework;  Knowledge  as 

Design 

Highly  structured  task  decompositions  (e.g.,  goal/task  analysis)  have 
been  widely  used  as  a  means  of  deriving  a  detailed  description  of  the 
domain  expert’s  behavior.  In  many  instances,  structured  task 
decompositions  have  been  utilized  as  the  sole  means  of  eliciting  domain 
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knowledge.  Although  such  techniques  generally  provide  means  of 
representing  an  important  aspect  of  the  domain  expert’s  interaction  with 
the  system,  they  fail  to  adequately  capture  the  domain  expert’s  perspective 
of  the  problem  space.  Because  such  techniques  merely  provide  a 
description  of  the  domain  expert’s  overt  behavior,  it  becomes  the 
responsibility  of  the  knowledge  engineer  to  interpret  the  behavior  and 
provide  the  cognitive  basis  for  its  occurrence.  These  highly  structured 
techniques  do  not  typically  elicit  the  expert’s  conceptualization  of  the 
problem  domain,  and  cannot  by  themselves  provide  an  adequate 
representation  of  the  domain  expert’s  knowledge. 

In  order  to  elicit  both  general  and  specific  knowledge,  and  succeed  in 
developing  a  large-scale  multidimensional  knowledge  base,  the  knowledge 
acquisition  technique  must  capture  more  than  simply  a  detailed  description 
of  the  behaviors  that  the  expert  exhibits  while  operating  in  the  specified 
problem  domain.  The  knowledge  acquisition  technique  must,  in  addition, 
be  capable  of  eliciting  the  problem  definition,  the  information  requirements 
and  the  expert’s  solutions  to  the  problem  from  the  domain  expert’s 
perspective. 

Structured  task  decomposition  and  the  method  used  to  elicit  the 
expert’s  conceptualization  of  the  problem  domain  provides  different  views  of 
the  user’s  expertise.  Yet,  the  transfer  of  knowledge  from  the  domain  expert 
to  the  knowledge  engineer  will  remain  incomplete  unless  it  also  includes 
an  exchange  of  information  which  has  enabled  the  domain  expert  to  make 
perceptual  discriminations  which  are  relevant  to  problem  solving  and 
felicitous  behavior  within  the  problem  domain.  As  a  person  develops 
expertise  in  a  given  domain,  he  or  she  comes  to  rely  upon  highly 
differentiated  patterns  or  attributes  perceived  from  the  environment  in 
order  to  be  able  to  successfully  guide  his  or  her  actions.3  Consequently,  as 
the  individual  begins  to  develop  expertise,  decisions  and  actions  are  based 
more  on  recognition,  and  less  upon  reasoning.4  Knowledge  acquisition 
techniques  must  go  beyond  the  acquisition  of  explicit  and  objective 
knowledge,  and  must  go  beyond  a  mere  verbal  description  of  an  expert’s 


1  This  will  not  be  true  of  all  problem  domains,  but  is  likely  to  be  especially  true  of  the 
tactical  fighter  environment. 

4  See  Klein  (1989)  for  a  discussion  of  recognition  based  decision  making. 
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pattern  recognition  processes.  It  is  not  enough  to  simply  encourage  the 
expert  to  verbalize  the  perceptually  salient  attributes  by  asking  the  domain 
experts  to  identify  the  perceptual  attributes  to  which  they  have  attended. 
Knowledge  that  has  been  derived  from  experience  rather  than  taught  using 
language  is  difficult  to  verbally  express.  Both  the  tacit  nature  of  the 
information  (Polany,  1966),  and  the  existence  of  a  mismatch  between  the 
knowledge  acquisition  context  and  the  context  during  which  the  perceptual 
learning  is  displayed  makes  it  unlikely  that  this  mere  encouragement  will 
facilitate  the  spontaneous  access  to  this  information. 

When  the  knowledge  engineer  tries  to  elicit  expert  knowledge  within 
the  confines  of  a  verbal  discourse,  the  expert  is  forced  to  rely  exclusively 
upon  mental  simulation  to  spur  recognition  of  experiential  knowledge. 

This  is  likely  to  severely  curtail  the  amount  and  accuracy  of  the  information 
that  is  elicited  as  the  expert  attempts  to  translate  tacit  knowledge  and 
perceptual  learning  into  a  verbal  domain.  When,  however,  the  knowledge 
acquisition  tools  provide  the  domain  expert  with  the  proper  medium  (i.e.,  a 
medium  more  closely  approximating  the  natural  context  in  which  the 
knowledge  is  used),  the  transfer  of  tacit  knowledge  and  perceptual  learning 
can  be  facilitated.  To  the  extent  possible,  a  knowledge  acquisition  technique 
should  provide  a  perceptual  context  which  can,  as  much  as  possible, 
simulate  the  conditions  under  which  the  domain  expert  would  typically 
make  use  of  the  perceptual  learning  that  he  or  she  has  accumulated.  With 
this  ecological  connection,  the  bottleneck  problem  will  be  reduced,  and 
much  of  the  intuitive  and  experiential  knowledge  will  be  spontaneously 
accessed. 

Although  the  pilot’s  conceptualization  of  mission  requirements  have 
been  woefully  absent  in  many  models  and/or  descriptions;  it  is  simply  not 
enough  to  create  a  framework  which  provides  only  this  view  of  the  problem 
domain.  The  information  that  is  provided  by  traditional  task  analyses  is 
also  important  in  so  far  as  it  represents  one  aspect  of  the  relationship  that 
will  exist  between  the  pilot  and  the  Pilot’s  Associate. 

The  inadequacies  of  traditional  language  based  knowledge  acquisition 
methods  has  prompted  the  inclusion  of  three  techniques  that  provide  the 
domain  expert  with  the  media  necessary  to:  1)  achieve  uninformed  access 
to  the  pilot’s  perspective  on  the  requirements  of  the  mission,  2)  acquire 
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access  to  the  analyst’s  perspective  on  the  mission  requirements,  and  3) 
elicit  the  tacit  knowledge  and  perceptual  learning  that  has  been  acquired 
through  experience  with  the  problem  domain.  Figure  1-2  presents  an 
overview  of  the  approach  taken  to  apply  the  advanced  user  requirement  and 
design  framework  to  the  PA/Pilot- Vehicle  Interface  problems  previously 
identified. 

A  more  complete  and  efficacious  framework  will  emerge  if  the 
potential  exists  to:  1)  capture  a  pilot’s  perception  of  his  mission;  2)  capture 
an  analyst’s  view  of  the  mission;  3)  integrate  both  these  perspectives  with 
the  designer’s  view  of  the  mission;  and  4)  provide  an  environment  where 
collaborative  analysis,  synthesis,  and  design  may  be  assimilated  and 
generated  in  exponential  fashion.  Each  of  these  perspectives:  the  pilot’s, 
the  analyst’s,  and  the  designer’s  is  elicited  using  a  different  procedure. 
Each  of  these  procedures  elicits  a  different  type  of  knowledge,  and  employs 
a  different  method  for  representing  that  knowledge.  The  primary  objective 
of  this  research  project  is  the  development  of  a  methodology  that  is  capable 
of  first  eliciting  the  knowledge  associated  with  each  of  these  perspectives, 
then  explicitly  representing  these  different  knowledge  types  using  different 
representational  techniques,  and  finally  proposing  a  ‘transformational 
grammar’  which  would  allow  for  the  establishment  of  interconnections 
among  these  different  knowledge  representations.  This  knowledge  and 
design  acquisition  methodology  is  intended  to:  1)  highlight  common  concept 
elements  for  analysis;  2)  identify  information  requirements;  3)  prototype 
and  evolve  design  storyboards;  and  4)  document  the  rationale  behind  any 
requirement  or  design. 

The  pilot’s  perspective  of  the  mission  and  his  or  her  experiential 
knowledge  is  captured  by  the  interactive  knowledge  elicitation  and 
representation  technique  termed  Concept  Mapping  (McFarren,  1987).  The 
emphasis  within  this  experiential  knowledge  representation  is  upon  the 
identification  of  requirements  and  the  associated  specific  knowledge  useful 
in  satisfying  these  requirements.  The  form  of  a  pilot’s  comprehension  of  a 
mission  may  be  declarative,  using  a  concept  definition  map,  or  procedural, 
using  a  time-line  concept  map.  In  either  form,  the  knowledge  elicitation  is 
derived  from  episodes  or  experience  which  each  pilot  can  access.  Within 
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either  form,  the  pilot’s  information  requirements,  decision-action  points, 
and  key  concepts  are  represented. 

The  concept  mapping  technique  offers  the  opportunity  to  access  two 
interrelated  processes  pertaining  to  the  pilot’s  conceptualization  of  the 
mission.  The  technique  offers  the  opportunity  to  acquire  the  ‘information 
heeded’  and  ‘information  remembered’  depending  upon  the  level  of 
intrusion  (i.e.,  probing)  that  occurs  during  the  elicitation  process.  The 
information  heeded  is  elicited  as  the  pilot  is  allowed  to  “think  aloud” 
wherein  there  is  minimal  amount  of  intrusion  by  the  interviewer.  The 
information  remembered  results  when  the  interviewer  intervenes  to  direct 
the  pilot’s  memory  along  a  selected  path. 

The  analytical  view  of  the  mission  elicits  a  structural  knowledge  in  the 
form  of  a  top-down,  normative,  functional  mission  decomposition.  This 
knowledge  is  captured  using  a  structural  analysis  tool  referred  to  as  either 
the  Structured  Analysis  and  Design  Technique,  (SADT),  or  the  Integrated 
Computer  Aided  Manufacturing  Definition  (IDEF).  The  emphasis  within 
the  structural  knowledge  representation  is  the  relationship  among  input, 
process,  control,  resources,  and  output  which  take  the  forms  of  mission 
procedures  (i.e.,  sequences  of  tasks),  tasks  (prescribed  actions  to  take),  and 
decisions  (selection  of  courses  of  action).  The  question  which  the  structural 
knowledge  element  of  the  overall  framework  purports  to  answer  is:  “What 
is  a  pilot  supposed  to  do  in  a  given  point  in  the  mission?”  (i.e.,  planning, 
monitoring  system  status,  performing  control  action,  etc.).  This  analytical 
knowledge  may  be  doctrinal  or  functional  in  nature,  but  engulfs  the  process 
of ‘informatioii  told’  as  formed  while  analyzing  a  mission  context. 

The  design  view  of  the  mission  elicits  ‘Knowledge  as  Design’  (see 
Perkins,  1986  for  more  specific  elaboration  on  this  topic)  usually  in  the  form 
of  visual,  tactile,  or  auditory  objects,  perceivable  as  prototypes  of  real  world 
referents.  Hence,  there  is  a  transformation  of  semantic-based  knowledge 
into  perceptual-object  based  (i.e.,  isomorphic)  representations  upon  which 
designers  tend  to  focus.  Knowledge  and  design  acquisition  can  now  involve 
the  noticing  of  perceptual  features  which  is  the  basis  for  problem  solving, 
discovery,  and  learning  (Bransford,  Sherwood,  Vye,  and  Rieser,  1986). 
Design  objects  are  prototyped  with  design  storyboarding  tools  (Andriole, 
1988)  using  the  Supercard  application  software  (Silicon  Beach  Software, 
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1990).  The  emphasis  within  the  knowledge  as  design  model  is  the  use  of 
specific  frames  which  unwind  over  time  to  provide  a  perceptual 
understanding  of  information  requirements,  for  a  targeted  event  or  mission 
segment.  The  design  storyboarding  process  first  involves  information  fused 
as  it  allows  a  pervasive  sweep  over  the  other  processes  involving 
information  heeded,  information  remembered,  and  information  told;  to 
synthesize  a  specific  design.  It  provides  the  knowledge  engineer  with  a 
means  of  answering  the  question:  “How  would  I  create  a  design  given  these 
requirements?”  Once  a  design  is  formulated  it  gives  rise  to  a  second 
process,  information  perceived,  which  affords  the  opportunity  for  other 
design  team  members  to  assess  and  evaluate  the  design  creation.  The 
question  that  the  design  storyboard  can  answer  is:  “What  do  I  perceive 
from,  and  how  should  I  interact  with,  the  given  design?”  When 
information  can  be  ‘perceived’  and  ‘fused’  iteratively,  new  information 
requirements  may  be  learned  and  new  facets  of  design  may  be  uncovered 
and  generated.  As  ‘designers’  begin  to  recognize  changes  in  their  own 
perceptions  of  a  problem,  learning  and  discovery  unfold.  Design  may 
simultaneously  be  recurrent,  concurrent,  and  doctrinal. 

The  knowledge  and  design  acquisition  methodology  that  will  be 
described  in  the  following  sections  provides  three  different  representations 
of  the  target  acquisition  phase  of  a  tactical  air-to-ground  mission.  The 
process  described  in  the  following  sections  is  intended  to  provide  a 
methodology  capable  of  eliciting  a  broad-based  understanding  of  the  pilot 
interaction  with  current  and  future  systems.  The  knowledge  and  design 
acquisition  methodology  begins  to  dissipate  the  traditional  knowledge 
acquisition  bottleneck  by  providing  a  medium  that  permits  the 
establishment  of  shared  understanding  between  the  knowledge  engineer 
and  the  domain  expert  which  is  both  verbal  and  perceptual  in  nature. 
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2  CONCEPT  MAPPING:  A  PILOT’S  VIEW  OF  THE  MISSION 
2.0  History  of  Concept  Mapping 

In  order  to  develop  a  theoretical  perspective  from  which  the  concept 
mapping  technique  can  be  considered,  a  brief  history  of  its  origins  will  be 
provided. 

2.0.1  Semantic  Networks 

The  origins  of  concept  mapping  can  be  traced  back  to  Quillian’s  (1968) 
seminal  work  on  semantic  networks.  The  semantic  network  model  was 
originally  proposed  as  a  representation  of  human  information  processing 
that  could  explain  some  of  the  numerous  effects  that  meaning  has  on 
memory.  It  was  intended  as  an  explanation  to  phenomena  such  as  the 
“category-size  effect”  in  which  it  was  found  that  category  classification 
takes  longer  when  the  item  is  a  member  of  a  large  category  class  than 
when  it  is  a  member  of  a  smaller  class.  Quillian’s  model  was  exceptionally 
simple,  consisting  of  points  (referred  to  as  nodes )  that  represented  concepts, 
and  the  arcs  between  the  points  representing  the  relationship  between  the 
concepts.  The  meaning  of  any  particular  concept  within  the  semantic 
network  was  represented  by  the  connections  (or  associations )  with  other 
concepts  within  that  network.  The  meaning  of  any  word  within  a  semantic 
network  was  expressed  by  its  relationships  to  other  words  resulting  in  what 
has  been  referred  to  as  concept’s  associative  structure. 

Collins  and  Quillian’s  (1969)  research  on  the  psychological  validity  of 
the  semantic  network  model  attempted  to  show  that  the  human  memory 
obeys  the  same  organizational  principles  that  are  exhibited  by  the  model. 
Specifically,  they  claimed  that  the  human  memory  obeys  the  organizational 
principles  of  hierarchy  and  economy.  The  first  organizational  principle, 
that  of  hierarchy,  asserts  that  semantic  memory  is  organized  in  a 
hierarchical  fashion.  The  validation  efforts  used  reaction  time  measures 
in  an  attempt  to  demonstrate  that  the  time  required  to  confirm  a  given 
proposition’s  validity  varied  as  a  function  of  the  number  of  inferential  steps 
between  the  concepts  included  within  the  proposition.  The  number  of 
inferential  steps  was  thought  to  increase  as  the  distance  between  the 
concepts  in  the  hierarchical  structure  increased.  For  instance,  Collins  and 


21 


Quillian  predicted  that  it  would  take  an  individual  longer  to  confirm  a 
proposition  like  A  canary  is  an  animal  than  it  would  to  confirm  a 
proposition  like  A  canary  is  a  bird  because  of  the  increased  number  of 
inferential  steps  through  the  hierarchy  to  confirm  the  former  proposition 
(see  Figure  2-1).  The  data  from  the  Collins  and  Quillian  study,  as  well  as 
the  data  from  similar  studies  (Conrad,  1972)  confirm  this  prediction. 

The  second  organizational  principle,  the  principle  of  economy,  asserts 
that  a  property  common  to  a  superset  concept  also  applies  to  those  nodes 
which  lie  beneath  it  in  the  hierarchy.  There  is  economy  produced  when  the 
attributes  are  stored  at  high  levels  and  not  stored  for  each  subordinate 
concept.  Leahey  &  Harris  (1985,  p  132),  for  example,  indicated  that 
information  such  as  “has  feathers”  and  “is  two-legged”  is  stored  at  the 
superordinate  node  “bird”  and  thus  need  not  be  stored  at  each  nodes 
subordinate  to  “bird”.  Collins  and  Quillian  (1969)  data  support  the  claims  of 
the  economy  principle;  however,  they  must  be  regarded  as  inconclusive,  in 
so  far  as  other  studies  (Conrad,  1972;  Smith,  Shoben,  &  Rips, 1974)  have  been 
able  to  generate  alternative  hypotheses  that  are  consistent  with  the  observed 
data. 

Although  the  empirical  tests  of  Quillian’s  (1968)  model  of  semantic 
memory  have  not  proven  entirely  conclusive  as  a  general  model  of  human 
memory,  Quillian’s  work  on  semantic  networks  continues  to  receive 
considerable  attention.  Later  models  of  human  semantic  memory 
(Anderson,  1976,  1980,  1989;  Anderson  &  Bower,  1973;  Hinton,  James,  & 
Anderson,  1989;  Kintsch,  1977;  Lindsay  &  Norman,  1977;  Minsky,  1986; 
Rumelhart,  Lindsay,  &  Norman,  1972;  Shank,  1984;  Sowa,  1984)  while 
showing  increasing  complexity,  tend  to  incorporate  as  the  underlying 
structure  a  semantic  network.  There  are  significant  differences  between 
these  various  accounts  of  human  semantic  memory,  but  there  does  appear 
to  be  a  considerable  degree  of  agreement  regarding  the  underlying 
structure  and  organization  of  semantic  memory,  a  structure  which  still 
bears  a  close  resemblance  to  Quillian’s  semantic  network.  The  basic 
premise  which  is  common  to  all  of  these  accounts  is  that  knowledge  is 
represented  by  concepts,  and  that  the  acquisition  of  additional  knowledge 
(i.e.,  learning)  is  based  upon  the  ability  to  take  the  basic  concepts  already 
possessed,  and  combine  them  as  needed  to  represent  any  additional 
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Figure  2-1  A  Hierarchical  Memory  Structure 


information  to  be  added  to  the  network.  Shank  (1984)  suggests  that  our 
ability  to  understand  information  is  a  consequence  of  being  able  to  relate  it 
to  what  we  already  know.  The  more  we  know  about  the  world  and  the  more 
experiences  we  have  had,  the  better  equipped  we  are  to  find  possible 
meanings  for  new  information.  Additionally,  the  more  we  know,  the  better 
we  comprehend  and  remember  new  knowledge  which  can  be  integrated 
within  prior  themes  (Morris,  Stein,  &  Bransford,  1979).  Similarly,  current 
theories  of  learning  suggest  that  knowledge  construction  and  elaboration 
occur  through  the  assimilation  of  key  ideas  from  a  complex  collection  of 
sources  into  an  interlinked,  meaningful  mental  model  (Spiro,  1977). 

Advances  in  semantic  network  models  have  currently  taken  the  forms 
of  schemata,  scripts,  and  neural  networks.  Consequently,  this  has 
broadened  the  issues  of  their  applicability  beyond  just  memory 
representation  to  now  include  the  use  of  memory  in  comprehension, 
meaning,  and  learning. 


2.1  Concept  Mapping  as  3  Teaching  Technique 

Developing  shared  knowledge  is  one  of  the  major  objectives  of  the 
education  process.  Research  on  learning  supports  this  idea  by 
demonstrating  that  students’  performance  improves  as  the  array  of 
associations  among  the  set  of  concepts  possessed  by  the  students  begins  to 
more  closely  approximate  the  array  of  associations  possessed  by  the  domain 
experts  (Brown,  Collins,  &  Duguid,  1989).  With  this  understanding  in 
mind,  it  is  widely  believed  that  the  structure  and  organization  of  human 
associative  memory  can  be  adequately  represented  using  semantic  network 
models.  Several  education  theorists  (Novak  &  Gowin,  1984;  Fisher,  Faletti, 
&  Quinn,  1990)  adopted  the  technique  for  graphically  representing 
information  that  is  to  be  transferred  from  one  individual  to  another  (see 
above  Figure  2-1).  This  method  provides  a  more  effective  way  of 
transferring  information  from  one  individual  to  another  by  overcoming 
many  of  the  limitations  inherent  in  the  linear  presentation  of  material. 

The  effectiveness  of  a  method  that  is  based  on  Quillian’s  semantic  network 
model  is  derived  from  the  fact  that  the  information  is  being  presented  in  a 
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form  that  more  closely  matches  the  cognitive  structure  of  the  individual 
acquiring  the  information  (Novak  &  Gowin,  1984;  Nosek  &  Roth,  1990).  In 
its  original  inception  this  representation  technique  is  referred  to  by  Novak 
and  Gowin  (1984)  as  concept  mapping. 

Concept  mapping  was  designed  as  a  method  that  would  be  consistent 
with  the  learner’s  cognitive  structure  and  which  would  externalize  for  both 
the  learner  and  the  teacher  ‘what  the  learner  already  knows.’  A  concept 
map  consists  of  two  or  more  concepts  that  are  linked  to  each  other,  thereby 
depicting  a  meaningful  relationship  that  exists  between  the  represented 
concepts.  The  concepts  within  this  concept  map  are  units  of  information 
such  as  objects,  phrases,  images,  sounds,  ideas,  and  events  which  are 
assigned  a  semantic  label.  Each  concept  is  understood  through  its 
relations  to  other  concepts.  The  relations  are  a  special  set  of  associations 
that  serve  to  describe  how  these  concepts  are  connected  to  one  another. 
Relations  are  linking  words  which  are  most  often  verbs  or  prepositions; 
however,  any  word  that  is  capable  of  expressing  the  relationship  between 
two  concepts  in  a  meaningful  fashion  can  function  as  a  relation.  The 
network  of  relations  gives  meaning  to  a  concept.  When  a  concept  map  has 
been  produced,  the  result  is  a  schematic  device  that  represents  a  set  of 
concept  meanings  embedded  in  a  framework  of  propositions  (Novak  & 
Gowin,  1984). 


2.2  Distinctions  between  Semantic  Networks  and  Concept  Maps 

Although  concept  maps  are  theoretically  related  to  semantic  networks, 
and  although  the  two  terms  are  often  treated  as  being  synonymous,  it  is 
useful  to  point  out  some  of  the  distinguishing  features  between  the  two  types 
of  representations.5  One  of  the  principle  distinguishing  features  between 
concept  maps  and  semantic  networks  is  that  concept  maps  are  constructed 
heterarchically  with  many  links  emerging  between  the  concepts,  and  with 


‘We  would  expand  codification  for  the  Pilot’s  Associate  program  to  also  include  design 
of  the  pilot-vehicle  interface  wherein  concept  maps  would  be  used  code  human  factor 
engineering  designs. 
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subordinate  concepts  in  the  array  being  linked  to  superordinate  concepts.6 
Semantic  network  models  (e.g.,  Quillian,  1968)  in  contrast  are  organized  in 
a  principally  hierarchical  fashion  with  relatively  few  links  between  the 
concepts.  In  this  sense,  concept  maps  may  be  more  closely  aligned  with 
connectionist  models  of  human  memory  (Rumelhart  &  McClelland,  1986) 
rather  than  the  earlier  semantic  network  models. 

Another  important  distinction  between  semantic  network  models  and 
concept  maps  is  that  the  semantic  network  models  are  typically  intended  to 
reflect  the  organizational  structure  inherent  to  human  semantic  memory 
in  general,  instead  of  reflecting  only  the  knowledge  and  experiences  of  the 
individual  whose  knowledge  is  being  graphically  represented.  The 
heterarchical  structure  which  emerges  in  the  concept  map  is  principally  a 
function  of  the  individual  whose  knowledge  is  leading  to  the  construction  of 
the  concept  map.  As  such,  the  meaning  embedded  within  a  concept  map  is 
a  product  of  an  individual’s  knowledge  and  experience  with  the  subject 
matter.  Because  of  this  fact,  any  individual’s  concept  map  is  necessarily 
subject  to  variations  over  time  as  additional  knowledge  is  incorporated  into 
an  existing  map,  and  as  the  context  against  which  the  information 
represented  within  the  concept  map  changes.  Such  changes  may  be  the 
result  of  assimilating  more  knowledge  over  time  (for  both  domain  expert 
and  knowledge  engineer)  as  well  as  recognition  by  the  domain  expert  that 
he  or  she  forgot  to  include  relevant  information  during  a  prior  session. 
Rather  than  being  indicative  of  how  the  human  mind  is  organized  in 
general,  or  being  representative  of  the  static  structure  of  that  organization, 
a  concept  map  is  a  snapshot  of  a  dynamic  process.  The  snapshot 
represents  the  way  the  individual  is  currently  thinking  about  the  concepts 
within  a  given  domain,  given  a  particular  context. 

As  the  context  from  which  an  individual  conceives  of  the  concepts 
changes,  the  concept  map  may  undergo  a  rearrangement  wherein  a 
subordinate  concept  assumes  a  superordinate  position,  causing  the 
subsequent  rearrangement  of  the  relationship  between  the  concepts.  This 
phenomenon  has  been  referred  to  as  the  “Rubber  Sheet”  effect  (see  Figure  2- 
2),  and  is  assumed  to  reflect  the  same  changes  that  occur  within  an 

‘It  should  be  reiterated  here  that  information  requirements  shown  on  this  summary 
map  are  not  inclusive,  but  merely  examples  and  provide  an  identification  of  areas  rich  for 
further  investigation. 
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Figure  2-2  A  Concept  Map  for  States  Showing  the  “Rubber  Sheet”  (adapted 
from  McFarren,  1987) 
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individual’s  mind  as  he  or  she  views  a  given  subject  matter  from  a  different 
perspective  or  from  amidst  a  different  context  (Novak  &  Gowin,  1984).  The 
same  concepts  are  portrayed  in  each  of  the  concept  maps  in  Figure  2-2,  but 
the  organization  has  been  changed  to  reveal  alternate  meanings  and  a  new 
emphasis  within  the  domain.  This  also  brings  up  the  point  that  whenever 
an  observer  views  a  concept  map,  it  is  from  that  person’s  personal  frame  of 
reference  which  is  not  necessarily  or  likely  to  be  the  same  as  the  domain 
expert’s.  Consequently,  different  people  see  different  things  in  concept 
maps  as  they  ‘mentally’  invoke  the  rubber  sheet  effect.  This  allows 
information  requirements  to  be  spawned  from  a  variety  of  different 
emphases.  The  network  of  relations  which  gives  meaning  to  the  concepts  is 
highly  context  dependent,  and  the  alternate  meaning  shown  in  the  two 
maps  is  depicted  through  the  change  in  the  links  that  exist  between  the 
concepts. 


2.3  Prior  Success  of  Concept  Mapping 

The  concept  mapping  technique  has  been  used  to  capture  the 
knowledge  structure  of  the  learner  in  order  to  facilitate  the  transfer  of 
information  from  the  teacher  to  the  learner.  With  the  representation  of  the 
student’s  knowledge  made  explicit,  it  becomes  easier  for  a  teacher  to 
indicate  to  the  student  the  relationship  that  should  exist  between  the  new 
concepts,  and  the  concepts  that  are  already  possessed  by  the  student 
(Armbruster  &  Anderson,  1984;  Novak,  Gowin  &  Johansen,  1983;  Novak  & 
Gowin,  1984).  The  technique  has  also  been  used  as  a  method  for  evaluating 
the  extent  of  the  student’s  knowledge  (Fisher,  Faletti,  &  Quinn,  1990; 
Naveh-Benjamin  et.  al.,  1986),  and  it  has  been  used  successfully  by  teachers 
to  convey  information  to  students.  Expert-produced  maps  have  been 
substituted  for  traditional  text  in  order  to  convey  complex  ideas  to  students 
with  positive  results  (Hall,  Dansereau,  &  Skaggs,  1988;  cited  in  Lambiotte 
et.  al.,  1989). 

Concept  mapping  has  proven  to  be  a  useful  technique  for:  1) 
transferring  information  from  one  individual  to  another;  2)  for  identifying 
the  key  ideas  within  a  given  subject;  3)  for  providing  a  formalism  that  is 
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closely  analogous  to  the  mental  organization  of  the  individual  being 
mapped;  and  4)  summarizing  a  given  cognitive  domain  (McFarren,  1987; 
Novak  &  Gowin,  1984).  Concept  mapping,  according  to  Fisher,  Faletti,  and 
Quinn  “lets  one  person  ‘see  what  another  is  thinking*  at  a  level  of  detail 
heretofore  unattainable.”  They  go  on  to  say  that  “this  formalism  is  (if 
semantic  networking  theory  is  correct)  more  analogous  to  mental 
representations  than  are  most  linguistic  structures,  (p.  4,  1990)”  and  thus, 
provides  the  researcher  and  educator  alike,  with  a  unique  window  into  the 
mind  of  the  individual  that  is  being  mapped.  Concept  maps  allow  one 
person  to  see  how  others  abstract  their  knowledge,  the  concepts  that  they 
choose  to  represent,  and  the  ways  in  which  they  choose  to  link  the  concepts 
together. 


2.4  Concept  Mapping  as  a  Knowledge  Acquisition  Tool 

Translating  the  concept  mapping  technique  from  a  tool  that  facilitates 
the  transfer  of  information  from  one  individual  to  another  in  an 
educational  setting,  to  a  tool  that  performs  similar  functions  for  the 
development  of  a  decision  support  system,  occurs  easily  and  without 
significant  modification  (McFarren,  1987).  When  used  as  a  knowledge 
acquisition  tool,  the  domain  expert  assumes  the  role  of  the  teacher,  and  the 
knowledge  engineer,  the  role  of  the  learner.  The  concept  map  that  is 
generated  constitutes  a  snapshot  representing  the  cognitive  structure  of  the 
domain  expert  and  can  be  used  to  transfer  this  information  to  the 
knowledge  engineer  and  to  the  designer  of  the  decision  support  system. 

The  information  that  the  knowledge  engineer  now  has  about  the  expert’s 
understanding  of  the  problem  domain  can  be  used  to  identify  the  user’ s 
requirement  for  a  decision  support  system  that  is  intended  to  aid  in  the 
performance  activities  within  this  specified  domain. 

The  same  characteristics  of  concept  mapping  that  make  it  an  effective 
method  for  transferring  information  in  an  educational  setting  make  it  an 
effective  technique  for  transferring  information  in  the  field  of  knowledge 
acquisition.  If  the  various  theories  regarding  the  structure  and 
organization  of  human  semantic  memory  are  correct,  including  some  of 
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the  current  theories  (e.g.  Anderson,  1980,  1989;  Hinton,  James,  & 

Anderson,  1989  Minsky,  1986;  Shank,  1984;  Sowa,  1984),  then  the  concept 
mapping  technique  offers  the  knowledge  engineer  a  tool  that  more  closely 
matches  the  characteristic  of  human  semantic  memory  than  other 
available  knowledge  acquisition  tools.  It  affords  a  way  of  escaping  the 
restrictiveness  of  the  linear  thought  process  that  is  reflected  in  both  written 
and  spoken  language  (Lambiotte,  et.  al.,  1989).  Concept  mapping  thereby 
allows  for  the  expression  of  the  domain  expert’s  knowledge  without 
requiring  that  it  be  translated  into  a  form  that  is  acceptable  for  typical 
communications. 

When  decision  support  systems  utilize  an  AI  architecture  and 
knowledge  representation  which  is  also  derived  from  semantic  networks 
(e.g.  scripts)  than  there  is  a  direct  transmittal  and  relationship  between 
human  memory,  the  acquisition  of  this  memory,  and  the  consequent 
representation  of  this  memory  in  an  AI  architecture.  This  then  becomes  a 
sound  basis  to  form  AI  models  which  can  be  highly  integrated  with  their 
operators  (see  McNeese,  1986). 

2.5  Concept  Mapping  Power 

Concept  mapping  is  an  interactive  interview  technique  which 
provides  both  the  expert  and  the  knowledge  engineer  a  medium  within 
which  to  communicate,  and  a  means  of  knowing  what  information  was 
communicated.  The  more  traditional  interview  techniques  do  not  readily 
allow  the  expert  to  know  what  the  interviewer  has  grasped,  nor  whether  it 
has  been  misinterpreted.  With  concept  mapping,  however,  the  expert  can 
see,  represented  on  the  board  before  him  or  her,  what  the  knowledge 
engineer  has  understood  or  what  the  knowledge  engineer  has 
misinterpreted.  This  unique  feature  of  the  concept  mapping  technique  also 
allows  the  domain  expert  to  quickly  correct  any  misrepresentations  of  his  or 
her  understanding  long  before  they  find  themselves  implemented  as 
software  or  hardware  designs. 

In  addition  to  capturing  the  domain  expert’s  knowledge  and  his  or  her 
understanding  of  the  problem  domain,  the  concept  mapping  technique  can 
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actually  aid  the  expert  in  organizing  the  presentation  of  his  or  her 
knowledge.  Because  the  expert  is  rarely,  if  ever,  asked  to  make  a  formal 
presentation  of  his  or  her  knowledge  domain  in  the  context  of  a  knowledge 
acquisition  session,  the  concept  map  provides  the  expert  with  a  way  of 
organizing  his  or  her  thoughts.  The  expert  can  freely  associate,  following  a 
particular  train  of  thought,  and  return  easily  to  the  original  idea  without 
losing  his  or  her  place  because  the  concept  map  serves  as  an  external 
memory  aid  that  shows  the  expert  where  he  or  she  has  been,  and  what  the 
interviewer  has  understood.  The  concept  mapping  technique  can  then  be 
used  by  the  domain  expert  to  assist  the  knowledge  engineer  and  system 
designer  in  understanding  the  nuances  of  a  knowledge  domain  and  help  in 
the  identification  of  areas  that  pose  performance  problems  for  the  domain 
expert. 

Concept  mapping  is  regarded  as  a  flexibly  non-obtrusive  knowledge 
acquisition  technique.  The  non-obtrusiveness  of  the  concept  mapping 
technique  refers  to  the  fact  that  the  technique  itself  does  not  interfere  with 
the  domain  expert's  conceptualization  of  the  problem  domain.  The 
technique  facilitates  a  consistency  between  the  expert's  understanding  of 
the  domain  and  the  way  in  which  that  information  is  represented.  The  idea 
of  flexibility  refers  to  the  fact  that  the  interviewer  can  probe  for  additional 
information  during  the  interview.  The  flexibly  non-obtrusive  quality  allows 
the  knowledge  engineer  to  vary  the  degree  to  which  he  or  she  directs  the 
course  of  the  mapping  session.  This  quality  also  affords  the  opportunity  to 
utilize  the  advantages  of  the  concept  mapping  technique’s  graphic  and 
interactive  attributes  with  other  strategies  for  eliciting  knowledge  from  a 
domain  expert.  For  instance,  the  concept  mapping  technique  can  be  used 
as  the  structure  within  which  the  Critical  Decision  Method  is  employed 
(Klein,  Calderwood,  &  Clinton-Cirocco,  1986;  Calderwood,  Crandall,  & 

Klein,  1987;  Klein,  Calderwood  &  MacGregor,  1989). 

The  concept  mapping  technique  facilitates  the  identification  of  key 
ideas  within  a  subject  area  through  the  use  of  its  graphical  representation 
techniques.  According  to  Lambiotte  et.  al.,  (1989)  “the  map’s  spatial 
properties  allow  the  individual  to  immediately  identify  characteristics  of  the 
knowledge  domain  such  as  overall  complexity,  differential  complexity  of 
subareas  of  the  map,  areas  of  symmetry  and  gaps  in  the  domain  expressed 
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by  breaks  in  symmetry,  continuation,  or  closure”  (p.  359).  This  automatic 
recognition  of  the  domain  characteristics  can  guide  the  map  reader  during 
the  extraction  of  detailed  meaning,  thereby  facilitating  the  identification  of 
key  ideas  within  the  particular  knowledge  domain.  The  graphical 
representations  employed  by  the  concept  mapping  technique  allow  for  easy 
comprehension  and  definition  of  complex  problem  spaces.  The  network  of 
concepts  that  are  portrayed  by  the  concept  map  enables  a  reader  of  the  map 
to  conceptualize  highly  complex  interrelationships  that  threaten  to  exceed 
human  cognitive  limitations. 

It  has  long  been  known  that  humans  possess  inherent  cognitive 
limitations  which  constrain  the  individual  to  actively  attend  to  no  more 
than  approximately  seven  units  of  information  (Miller,  1956).  Although  it 
cannot  be  said  that  a  concept  map  permits  an  individual  to  exceed  this 
limit,  it  does  have  the  characteristics  which  enable  an  individual  to  work 
more  effectively  within  the  limitations  that  do  exist.  The  graphic 
characteristic  of  the  concept  map  will  allow  the  reader  to  use  an  external 
memory  aid  in  order  to  grasp  complex  interactions  among  concepts  that 
could  otherwise  potentially  exceed  his  or  her  cognitive  capacity.  The 
concept  map  also  allows  readers  the  opportunity  to  ‘chunk’  together  concept 
clusters  to  effectively  expand  the  size  of  units;  thereby,  increasing  the  range 
of  their  cognitive  capacity. 


2.6  Practical  Issues  Affecting  Knowledge  Acquisition 

In  addition  to  being  well  grounded  in  theory,  it  is  important  that 
knowledge  acquisition  techniques  also  satisfy  several  practical  concerns  in 
order  for  the  method  to  actually  be  effective  at  eliciting  knowledge  from  a 
domain  expert.  As  Klein,  Calderwood,  and  MacGregor  (1989)  point  out,  any 
method  that  is  to  be  at  all  useful  must  first  satisfy  certain  basic 
requirements  with  regard  to  practicality. 

First,  the  method  must  be  time  efficient,  as  it  is  unusual  to  be  able  to 
secure  more  than  a  two-hour  interview  session  with  any  given  domain 
expert.  Even  if  greater  amounts  of  time  were  available,  the  amount  of 
information  that  can  be  elicited  from  an  individual  begins  to  rapidly  wane 
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as  fatigue  sets  in.  Thus,  the  knowledge  acquisition  technique  must  make 
efficient  use  of  the  time  that  is  available,  and  be  flexible  enough  to  allow  for 
the  integration  of  several  short  knowledge  acquisition  sessions.  Concept 
mapping  makes  efficient  use  of  the  domain  expert’s  time  by  facilitating  the 
rapid  transfer  of  domain  knowledge  and  enabling  the  knowledge  elicitor  to 
prepare  and  prioritize  probes  that  can  direct  the  expert’s  presentation  of 
information.  The  graphic  presentation  method  that  is  employed  with  the 
concept  mapping  technique  makes  the  technique  particularly  well  suited 
for  use  during  extremely  short  knowledge  acquisition  sessions.  The 
domain  expert  will  be  able  to  quickly  identify  where  he  or  she  has  stopped 
during  the  previous  session,  thereby  highlighting  its  ability  to  establish 
consistency  and  continuance  across  sessions. 

Second,  the  method  must  provide  a  cost  effective  means  for  data 
collection  and  analysis.  With  budget  restrictions,  it  is  important  that  the 
collection  of  domain  knowledge  not  be  cost  prohibitive.  Collecting  and 
analyzing  a  speak-aloud  protocol  generated  by  running  the  domain  expert 
through  a  veridical  simulation  of  the  domain  environment,  may  provide  a 
rich  source  of  information.  Unfortunately,  the  costs  associated  with  such 
procedures  are  significant.  Also,  the  depth  and  breadth  of  the  knowledge 
acquired  may  not  be  better  than  those  of  knowledge  gleaned  using  the 
knowledge  and  design  acquisition  techniques  that  are  being  described  in 
this  report.  In  contrast  to  a  full  scale  simulation  and  all  of  its  associated 
costs,  the  concept  mapping  technique  requires  nothing  more  than  a  writing 
surface  (chalk  board,  dry-marker  board,  paper,  etc.)  and  a  tape  recorder. 

Third,  the  knowledge  acquisition  method  must  be  capable  of 
representing  the  knowledge  that  has  been  gleaned  from  the  knowledge 
acquisition  session  in  a  form  that  facilitates  its  codification  in  a  decision 
support  system  (Klein,  1990).  This  process  frequently  involves  a  translation 
of  the  information  derived  from  the  knowledge  acquisition  sessions  into  a 
form  that  allows  for  easy  communication  between  the  parties  involved  in 
the  process  of  design  and  constructing  decision  support  systems.  The 
translation  can  be  a  time  consuming  and  difficult  process,  and  perhaps 
more  significantly,  it  can  involve  the  loss  or  misinterpretation  of 
information  as  it  is  changed  from  one  representational  medium  into 
another.  The  concept  mapping  technique  provides  a  distinct  advantage  in 
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so  far  as  the  domain  expert’s  knowledge  is  captured  during  the  knowledge 
acquisition  session  in  a  form  that  can  be  directly  embodied  in  a  decision 
support/AI  system  (e.g.,  Bareiss,  1989;  Carbonell,  1970;  Buchanon  & 
Shortliffe,1984;  Duda,  Hart,  Barrett,  Gaschnig,  Konolige,  Reboh,  &  Slocum, 
1978;  Duda,  Hart,  Nilsson,  &  Sutherland,  1978;  Lebowitz,  1986;  Nirenburg, 
Monarch,  Kaufmann,  Nirenburg,  &  Carbonell,  1988;  Routh,  Milne,  & 
Kabrisky,  1986).  The  representation  of  expert  knowledge  in  a  semantic 
network  is  useful  because  it  facilitates  “making  deductions  about 
inheritance  hierarchies,  and  .  .  .  enables  more  direct  and  controlled  search 
rather  than  a  search  through  the  whole  data  base”  (Nosek  &  Roth,  1990). 

Nosek  and  Roth  (1990)  recently  conducted  a  study  in  which  they 
compared  what  they  regarded  as  the  two  most  popular  knowledge 
representation  techniques  currently  being  used  in  the  AI  field,  semantic 
networks  and  predicate  logic.  Nosek  and  Roth’s  work  indicates  that  a 
semantic  network  scheme  is  a  transparent  method  for  representing  the 
expert’s  knowledge  in  the  sense  that  its  structure  does  not  interfere  with  an 
individual’s  ability  to  comprehend  the  information  that  is  being 
represented.  Specifically,  the  results  of  their  study  reveal  that  the  semantic 
network  scheme  as  a  knowledge  representation  technique  was  superior  to 
predicate  logic  in  the  areas  of  problem  identification,  comprehension, 
generalization  and  application.  However,  while  Nosek  and  Roth  recognize 
the  usefulness  of  semantic  network-like  representations  in  facilitating  the 
communication  of  information  between  the  knowledge  engineer  and  the 
domain  expert,  they  conceive  of  it  as  primarily  a  method  for  validating  the 
transfer  of  information  without  considering  its  potential  utility  as  a  method 
for  eliciting  the  information.  According  to  Nosek  and  Roth: 

“The  knowledge  of  the  expert  is  transferred  to  the  knowledge 
engineer  through  the  communication  channels  of  oral  and 
written  descriptions  and  through  observation.  To  obtain  the 
expert’s  validation  of  the  transfer  process  from  the  expert  to  the 
knowledge  engineer,  the  knowledge  engineer  transfers  the 
knowledge  content  to  the  expert  via  a  representation  scheme.  The 
representation  scheme  then  becomes  the  major  communication 
vehicle”  (p.  228,  1990). 

We  are  suggesting  that  the  utility  of  representation  schemes  can  be  greatly 
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enhanced  if  the  method  is  also  used  as  the  medium  for  transferring  the 
knowledge  of  the  expert  to  the  knowledge  engineer  (in  the  form  of  the 
concept  mapping  technique)  instead  of  only  serving  as  a  technique  for 
validating  the  original  communication  which  has  been  accomplished  using 
other  knowledge  acquisition  techniques. 

The  fourth  practical  issue  affecting  the  selection  of  knowledge 
acquisition  methods  involves  the  level  of  training  needed  by  the  knowledge 
engineer.  The  nature  of  the  concept  mapping  technique  is  such  that  it 
requires  a  minimal  amount  of  prior  domain  knowledge  in  order  for  the 
knowledge  engineer  to  effectively  use  the  method.  In  addition,  the 
technique  itself  is  easy  to  master  (for  both  the  knowledge  engineer  and  the 
domain  expert)  thereby  avoiding  lengthy  training  sessions. 


2.7  Concent  Mapping  Basics;  Methodology.  Syntax  and  Structure 

The  use  of  concept  mapping  as  a  knowledge  acquisition  tool  involves 
interactively  producing  a  concept  map  of  the  domain  expert’s  knowledge 
during  an  interview  with  that  expert.  The  concept  map,  in  fact,  becomes 
the  vehicle  for  transferring  knowledge  from  the  domain  expert  to  the 
knowledge  engineer  as  the  domain  expert  leads  the  knowledge  engineer  on 
a  conceptual  journey  through  the  subject  material.  Unlike  other  interview 
techniques,  whether  structured  or  unstructured,  the  concept  mapping 
technique  enables  the  domain  expert  to  literally  see  what,  and  how  the 
knowledge  engineer  has  interpreted  the  information  that  he  or  she  has 
presented.  A  frequent  complaint  issued  by  domain  experts  concerning  the 
use  of  interviews  for  knowledge  acquisition,  is  that  the  knowledge  engineer 
‘hears  and  remembers  only  what  is  consistent  with  his  or  her  prior 
conception’  and  not  what  the  domain  expert  actually  intended.  Regardless 
of  the  reasons  resulting  in  this  misrepresentation  of  information,  the 
interactive  nature  of  the  concept  mapping  technique  enables  the  domain 
expert  to  review  and  correct  misrepresentations  as  they  occur  (as  well  as  at 
a  later  date  after  the  maps  have  been  cleaned  up  by  the  knowledge 
engineer).  Hence,  the  domain  expert  is  involved  in  both  on-line  and  off-line 
review  of  the  map. 
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In  order  to  ensure  that  the  concept  mapping  session  remains 
interactive,  it  is  necessary  that  the  map  be  legible  to  both  knowledge 
engi:^er  and  domain  expert.  The  map  that  is  produced  during  the  session 
closely  resembles  the  map  shown  above  in  Figure  2-1.  The  form,  structure, 
and  syntax  of  the  concept  maps  used  for  the  purposes  of  knowledge 
acquisition  are  kept  extremely  simple  in  order  to  ensure  that  the 
representation  is  transparent  to  a  reader  (McFarren,  1987).  The  graphic 
representation  consists  minimally  of  a  pair  of  concepts,  which  are  assigned 
a  semantic  label  and  linked  to  one  another  by  the  relationship  existing 
between  the  two  concepts,  (see  Figure  2-3).  As  the  complexity  of  a  given 
problem  domain  increases,  additional  relations  and  concepts  are  added  to 
the  map  without  requiring  an  increase  in  the  complexity  of  the  syntax  that 
is  used  to  govern  the  structure  of  the  map. 

Throughout  concept  mapping’s  history  of  usage  in  both  educational 
and  computer  science  settings,  there  has  been  a  tendency  to  increase  the 
number  of  rules  governing  the  construction  of  concept  maps  in  the  belief 
that  this  increased  specificity  will  lead  to  greater  precision  (Lambiotte  et. 
al.,  1989).  For  instance,  efforts  have  been  made  to  define  a  canonical  set  of 
relations  that  could  be  used  to  parsimoniously  represent  the  knowledge  in  a 
given  domain  (Holley  &  Dansereau,  1984;  Fisher,  et.  al.,  1990).  Lambiotte, 
et.  al.  describe  a  wide  array  of  graphical  conventions  that  they  have  adopted 
in  the  belief  that  such  techniques  will  foster  the  efficient  communication  of 
information  about  concepts  and  the  multiple  relationships  among  the 
concepts.  While  these  efforts  may  in  fact  promote  parsimony  with  respect 
to  the  analysis  of  the  concept  map,  they  were  viewed  as  counter-productive 
in  the  context  of  the  goals  of  a  knowledge  acquisition  technique.  Our  use  of 
the  concept  mapping  technique  has  led  us  to  believe  that  several  of  the 
technique’s  principle  advantages  as  a  knowledge  acquisition  tool  (i.e.,  its 
use  in  maintaining  an  interactive  interview,  and  its  transparency  to  the 
reader)  would  be  compromised  if  these  various  additional  rules  were 
actually  incorporated.  The  one  rule  that  should  be  followed  is  to  insure  that 
all  the  relations  and  concepts  are  labeled,  for  it  has  been  found  that  maps 
that  have  been  produced  without  labeled  arcs  are  difficult  to  comprehend 
(Lambiotte,  et.  al.,  1989). 
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Figure  2-3  Concept  Mapping  Syntax 


2.8  Concept  Mapping  of  a  Tactical  Air-To-Ground  Mission 


This  section  describes  the  applications  of  the  concept  mapping 
technique  to  the  problem  of  eliciting  domain  knowledge  from  experienced 
tactical  fighter  pilots  for  the  purposes  of  identifying  and  validating  a 
detailed  set  of  user  requirements.  This  methodology  allows  for  not  only  the 
identification  of  user  requirements,  but  also  facilitates  the  communication 
of  user  requirements  to  the  designers  of  the  Pilot’s  Associate.  It  should  be 
noted,  however,  that  the  present  effort  is  intended  to  be  a  demonstration 
that  these  methods,  if  applied  in  a  full  scale  effort,  would  be  a  highly 
effective  means  of  capturing  the  user’s  understanding  of  the  problem 
domain.  Thus,  the  results  presented  should  not  be  considered  definitive, 
but  rather  indicative  of  the  type  of  information  that  this  technique  is  capable 
of  generating.  In  order  to  facilitate  this  investigation,  the  concept  mapping 
sessions  focused  only  on  the  target  acquisition  phase  of  the  tactical  fighter 
mission,  and  were  directed  toward  uncovering  the  pilot’s  information 
requirements  for  a  pilot-vehicle  interface. 


2.9  Methodology 

The  interview  sessions  were  conducted  using  an  interview  panel 
format  with  one  domain  expert,  and  two-to-five  interviewers  assisting  with 
the  interview.  A  single  interviewer  served  as  the  concept  mapper  during 
all  of  the  interview  sessions.  The  other  interviewers  participating  in  the 
session  were  tasked  with  generating  the  questions  and  probes  in  order  to 
gain  more  detailed  information  about  the  concepts  that  were  being 
presented  by  the  domain  expert. 

Eight  domain  experts  were  interviewed  during  the  concept 
demonstration  phase  of  this  project.  Six  of  these  were  tactical  fighter  pilots 
with  an  average  of  2300  hours  of  logged  flight  time  (ranging  between  700 
and  5000  hours)  in  F-4s,  and  F-16s,  and  extensive  experience  flying  tactical 
air-to-ground  missions.  Of  the  remaining  two  domain  experts,  one  was  a 
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F-4  backseater  and  pilot  instructor,  and  the  other  was  a  B-52  pilot  with 
extensive  tactical  air-to-ground  experience. 

Each  domain  expert  was  interviewed  for  an  average  of  two  hours  on 
two  separate  occasions,  with  the  focus  of  the  session  differing  on  each 
occasion.  The  first  round  interviews  were  intended  to  elicit  general 
knowledge  of  the  domain  that  was  independent  of  a  specific  aircraft, 
mission,  target,  or  weapon  type.  This  session  was  divided  into  two  parts  of 
approximately  equal  length.  The  first  half  of  the  interview  concentrated  on 
the  definition  of  target  acquisition  during  a  tactical  air-to-ground  mission, 
and  the  second  half  focused  on  the  specific  procedures  and  decisions  that 
the  pilot  would  make  during  that  particular  phase  of  the  mission.  The 
second  round  interviews  were  similarly  divided  into  two  parts,  with  the 
first  half  focusing  on  the  validation  and  clarification  of  the  map(s)  produced 
during  the  first  session  interview.  During  the  second  half  of  the  second 
interview  session,  the  pilots  were  given  a  specific  mission  profile  consisting 
of  a  specified  target,  weapon  type  and  attack  geometry  in  order  to  establish 
the  context  for  the  interview.  With  this  context  as  a  backdrop,  the  pilots 
were  probed  for  additional  detailed  knowledge  regarding  the  key  decision 
points  encountered  during  the  target  acquisition  phase  of  the  mission,  and 
the  information  used  to  make  the  decisions. 

The  first  session  interview  began  with  a  brief  introduction  to  the 
concept  mapping  technique.  Because  this  knowledge  elicitation  technique 
is  an  interactive  one,  it  was  important  that  the  domain  expert  (i.e.,  the  pilot) 
had  at  least  a  rough  understanding  of  what  the  technique  was,  and  what  he 
was  expected  to  contribute.  It  was  explained  to  the  pilot  that  the  technique 
that  was  being  used  was  designed  to  capture  his  understanding,  and  that 
we  were  primarily  concerned  with  his  conceptualization  of  the  problem 
domain  (i.e.,  tactical  air-to-ground  mission).  In  order  to  insure  that  the 
knowledge  depicted  in  the  concept  map  actually  represented  the  pilot’s 
understanding  of  the  problem  domain,  the  pilot  was  strongly  encouraged  to 
correct,  edit,  or  re-draw  the  map  that  was  being  produced.  The  pilot  was 
also  informed  at  this  time  that  an  audio  recording  of  the  session  would  be 
produced. 

The  context  for  this  knowledge  acquisition  session  was  set  by 
informing  the  domain  expert  of  the  specific  portion  of  the  problem  domain 
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of  interest.  Specifically,  the  pilots  were  informed  of  our  interest  in  the 
target  acquisition  phase  of  an  air-to-ground  mission.  The  pilots  were  asked 
to  describe  the  procedures  and  decisions  involved,  and  we  stressed  the  fact 
that  we  were  particularly  interested  in  the  information  that  they  use  to 
make  each  decision  and  how  this  information  should  be  displayed.  The 
pilots  were  also  asked  to  describe  the  various  concerns  that  they  have 
during  each  segment  of  the  target  acquisition  phase  of  the  mission.  During 
the  first  interview  session,  few  constraints  were  imposed  upon  the  focus  of 
the  discussion,  beyond  limiting  it  to  the  target  acquisition  phase  of  tactical 
air-to-ground  mission.  In  other  words,  the  pilot’s  were  not  asked  to  limit 
their  discussion  to  a  specific  aircraft,  mission,  target,  or  weapon  type.  The 
rationale  for  avoiding  the  imposition  of  excessive  constraints  was  to  allow  a 
wide  exposure  of  the  issues  during  the  initial  round  of  interviews. 

Throughout  the  mapping  session,  the  pilot's  conceptualization  of  the 
target  acquisition  phase  of  the  mission  was  captured  in  the  concept  map. 

As  the  pilot  presented  information,  a  concept  map  was  drawn  on  a  white, 
dry-marker  board  in  front  of  him  so  that  both  he  and  interviewers  could 
easily  see  and  discuss  the  concepts  that  were  being  represented.  Typ  cally, 
the  map  was  drawn  by  one  of  the  interviewers  as  the  pilot  spoke,  although 
on  several  occasions  the  pilots  have  themselves  participated  in  the  drawing 
of  their  own  concept  map.  The  fact  that  the  map  was  drawn  in  front  of  the 
domain  expert  so  that  he  could  read  the  map  as  it  emerged  is  a  particularly 
important  aspect  of  the  interview  process,  and  the  pilots  were  encouraged  to 
interact  with  the  maps  by  reading,  editing  and  correcting  them  throughout 
both  portions  of  the  concept  mapping  session. 

At  appropriate  times  throughout  the  mapping  session,  the  pilot  was 
probed  with  questions  (both  planned  and  impromptu)  which  asked  for 
clarification  or  additional  information  on  concepts  that  were  being 
discussed  (see  Table  2-1).  A  conscientious  effort  was  made  not  to  interrupt 
the  pilot’s  train  of  thought,  but  to  ask  the  question  only  when  additional 
information  was  sought  regarding  a  particular  concept,  or  when  there  was 
a  lull  in  the  discussion.  Thus,  in  practice,  the  majority  of  the  questions 
were  raised  after  the  pilot  had  concluded  his  presentation  of  his 
understanding  of  the  target  acquisition  phase  of  a  tactical  air-to-ground 
mission.  Because  the  concept  map  provides  a  clear  graphical 
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TABLE  2-1. 


Potential  Probes  For  Concept  Mapping  Session 


1)  Define  and  elaborate  upon  concepts  associated  with: 

Attack 

Weapon  Release 
Changes  in  Target  Location 
Attack  Plan  Deviation 
Target  Misidentification 
G-Limit/Heavyweight  Maneuvering 

2)  How  is  target  acquisition  effected  by: 

Weapon  Type 
Target  Type 
Weather 

Terrain/Obstacles  Obscuring  Target 
Pop-up  Threats 
Equipment  Malfunctions 

3)  What  do  you  look  for  when  you  are  trying  to  find  the  target 
area,  and  target  just  prior  to  weapons  release? 

4)  What  is  the  most  difficult  aspect  of  target  acquistion  during 
an  air-to-ground  mission? 

5)  When  describing  the  steps  involved  in  target  acquisition  you 
mention  transitioning  from  one  step  or  sequence  to  another, 
what  information  do  you  use  when  deciding  to  move  on  to 
the  next  step? 

6)  What  is  the  source  of  the  information  that  you  use  when 
making  the  decision?  How  is  that  information  displayed? 
How  would  you  like  to  see  it  displayed? 

7)  What  information  would  have  aided  the  decision,  making  it 
more  timely,  more  accurate,  less  demanding  or  less 
uncertain? 

8)  What  are  your  goals  during  this  particular  phase  of  the 
mission? 

9)  What  are  your  principle  concerns  during  this  particular 
phase  of  the  mission? 
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representation  of  the  pilot’s  knowledge,  the  questions  could  be  asked  in 
such  a  way  that  they  referred  back  to  a  particular  portion  of  the  concept 
map  thereby  allowing  for  the  integration  of  the  additional  information  with 
the  existing  map. 

During  the  second  phase  of  the  first  round  interview,  when  the  specific 
procedures  and  decisions  that  the  pilot  makes  during  the  target  acquisition 
phase  were  being  discussed,  an  event  time-line  was  provided  to  facilitate 
the  mapping  of  sequential  processes  as  well  as  the  pilot’s  thoughts 
concerning  those  processes.  The  time-line  also  afforded  a  means  to 
represent  the  specific  context  against  which  a  particular  sequence  of 
activities  would  be  cast  (see  Figure  2-4). 

After  the  completion  of  the  second  phase  of  this  interview,  the  pilot  was 
given  the  opportunity  to  review  the  map  that  had  been  generated.  Even 
though  the  pilot  had  been  interacting  with  the  map  throughout  the  course 
of  its  generation,  he  was  given  an  additional  opportunity  to  correct  any 
misrepresentations  and  to  fill-in  any  omitted  information. 

The  second  session  interviews  began  with  a  review  of  the  concept  maps 
that  were  produced  during  the  previous  interview  with  that  pilot.  A  large 
scale  reproduction  of  the  computer  rendered  maps  served  as  the  focus  of  the 
first  phase  of  the  second  session  interview.  The  maps  were  reviewed  by  the 
interview  team  and  the  specific  areas  that  required  either  clarification  or 
additional  detail  were  noted  prior  to  initiation  of  the  second  session  of  the 
interview.  The  pilot  was  given  the  opportunity  to  make  corrections  in  the 
map  both  before  and  during  the  second  session  interview.  When  the  pilot 
was  satisfied  with  the  accuracy  with  which  his  knowledge  of  the  target 
acquisition  phase  of  a  tactical  air-to-ground  mission  had  been  represented, 
the  interview  changed  its  focus. 

During  the  next  phase  of  the  interview,  the  pilots  were  provided  with  a 
specific  mission  profile  which  identified  the  target  type,  weapon  selection, 
attack  geometry  and  potential  threat  encounters  (see  Table  2-2). 

With  the  mission  profile  serving  as  the  context  for  the  remainder  of  the 
interview,  the  pilots  were  next  asked  to  verify  the  accuracy  and 
completeness  of  a  list  of  decision  points  between  the  Initialization  Point  (IP) 
and  Weapon  Release  Point  (WRP)  of  the  mission.  Once  completed,  a 
detailed  concept  map  was  produced  for  each  decision  point  which  focused 
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Figure  2-4  Time-line  Concept  Map  Showing  a  Portion  of  a  Tactical  Fightor  Mission 


TABLE  2-2. 


Mission  Profile  for  Second  Concept  Mapping  Session 


Mission:  A  two-ship  attack  for  an  interdiction  mission.  The 
target  area  is  approximately  100  miles  beyond  the  FEBA. 

Target:  Destroy  2  POL  storage  tanks  30  feet  diameter  x  24  feet 
tall,  separated  by  350  feet,  and  if  possible  neutralize  pumping 
station.  The  tanks  contain  one  of  the  two  agents  used  for 
binary  chemical  weapon,  it  is  not  especially  toxic  until  mixed, 
but  is  very  volatile.  A  large  secondary  explosion  is  to  be 
expected  with  fragmentation  up  to  5000  feet  vertical  and  9000 
f  et  horizontal.  Fragments  are  predicted  to  be  back  on  the 
ground  after  approximately  30  seconds.  The  target  area  is  a 
rail  depot  approximately  four  miles  west  of  a  moderately  large 
city. 

Weapon  Selection:  Munitions  consists  of  four  CBU-87’s  per 
aircraft  (CBU-87  contains  202  armor  penetrating  incendiary, 
blast  fragmenting  bomblets).  In  addition,  two  Sidewinder 
missiles  will  be  carried  by  the  aircraft  in  order  to  provide 
protection  in  an  Air-to-air  environment. 

Time  Over  Target:  The  time  over  target  for  this  mission  is  0700 
-0715 

Weather:  The  weather  for  both  take  off  and  landing  is  forecast 
to  be  clear  with  unlimited  visibility.  The  weather  in  the  target 
area  will  be  limited  to  a  ceiling  between  12,000  and  8,000  feet 
AGL,  with  visibility  limited  by  10  to  20  %  cloud  cover.  Winds 
are  variable  at  5  to  10  Kts. 

Terrain:  Just  over  one  hundred  miles  are  planned  at  low 
altitude.  Low  level  flight  during  this  portion  of  the  mission  will 
be  over  flat  arid  land  for  approximately  half  of  the  time,  and 
over  mountainous  terrain  for  the  remaining  distance.  The 
terrain  from  IP  to  target  will  be  mountainous  with  the  target 
being  located  in  gently  sloping  terrain. 
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Table  2-2,  continued 


Ground  Threats:  The  flight  will  potentially  be  within  the 
effective  range  of  long  range  search  radars  throughout  the 
mission.  SA-6  and  SA-8  are  known  to  be  located  in  the  target 
area,  with  SA-10  and  SA-14’s  along  the  egress  route.  Anti¬ 
aircraft  artillery  may  also  be  encountered  during  ingress,  and 
IP  to  target  run-in. 

Air  Threats:  The  initial  air  threat  is  posed  by  MiG-29’s  that 
are  operating  along  the  FEBA,  with  a  possibility  of  MiG-23  and 
MiG-25’s  in  the  target  area. 

Timing:  The  timing  for  this  mission  ,  while  not  especially 
critical,  should  be  adhered  to  as  closely  as  possible.  This  is 
expected  to  minimize  likely  threats,  and  apply  ordnance  when 
target  is  most  vulnerable. 

C&  The  command  and  control  for  this  mission  will  be  provided 
by  an  ABCCC  with  appropriate  inputs  from  JSTARS. 

Weapon  Delivery  Method:  Double  90, 10°  LAB,  with  a  30  sec 
delay  between  flights 

Attack  Geometry: 

Pull  Down  Altitude:  1500  ft  (AGL) 

Apex  Altitude:  2300  ft 

Planned  Release  Altitude:  1100  ft 

Minimum  Release  Altitude:  900  ft 
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on  the  information  that  is  used  by  the  pilot  to  recognize  that  a  decision  point 
has  been  reached  as  well  as  the  specific  information  that  the  pilots  utilize  to 
make  the  decision.  The  pilots  were  probed  at  this  point  for  details 
pertaining  to:  1)  the  current  source  of  the  information  (i.e.,  the  avionic 
system/sensor  that  was  used);  2)  possible  improvements  in  the  way  in 
which  the  information  is  presented;  3)  areas  where  their  performance 
could  be  enhanced  with  the  inputs  from  a  Pilot’s  Associate;  4)  ways  a  Pilot’s 
Associate  should  interact  with  the  pilot;  and  5)  possible  ideas  about  function 
allocation  tradeoffs.  The  information  derived  from  these  probes  was 
integrated  directly  into  the  concept  map  that  was  being  drawn. 


2.10  Synthesis  of  Summary  Mans 

Upon  completion  of  each  interview,  an  extensive  map  review  process 
was  begun.  The  first  step  in  the  process  involved  a  review  of  the  audio  tape. 
During  this  review,  a  comparison  of  the  information  contained  on  the  tape 
was  made  with  the  information  that  had  been  represented  with  the  concept 
map  to  insure  that  all  pertinent  information  was  completely  and  accurately 
represented.  Once  the  review  was  completed,  the  interview  session  map 
was  transcribed,  and  then  redrawn  using  a  computer-aided  drafting 
application.  The  process  of  computer  rendering  the  concept  maps  proved  to 
be  lengthy  and  time  consuming  which  nevertheless  was  deemed 
worthwhile  because  of  the  intention  to  use  the  maps  produced  during  the 
first  interview  as  the  basis  for  the  second  interview.  During  the  process  of 
transcription  from  the  hand-drawn  to  the  computer-rendered  map,  the 
content  of  the  maps  was  examined  and  subsequently  edited  in  order 
to  insure  that  all  ‘node-link-node’  units  were  inherently  meaningful,  and 
that  the  map  was  free  of ‘sentence  mappings 7  (see  Figure  2-5). 

There  is  no  uniform  way  to  map  a  particular  domain,  and  the  maps 
that  various  pilots  produce  are  personal  expressions  of  their  ‘thinking’  on 
the  issue  of  target  acquisition  at  a  level  of  detail  not  previously  available. 

1  Note,  that  although  the  viewpoint  is  said  to  be  that  of  the  pilot’s,  it  does  not  serve  to  alter 
the  perspective  that  is  represented  using  this  technique.  The  analyst,  in  this  case,  is 
attempting  to  assume  the  pilot’s  viewpoint  during  the  performance  of  a  tactical  air-to- 
ground  mission,  and  thus,  represents  the  analyst’s  perspective  of  the  user  requirements. 
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"Sentence  Map" 


Concept  Map 


Figure  2-5  Sentence  Map 
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This  particular  problem  was  confronted  during  the  task  of  summarizing 
several  individual  maps  into  a  single  composite  map.  However,  since  we 
are  particularly  interested  in  the  global  pattern  behavior,  and  common 
ways  of  thinking  about  that  behavior,  it  becomes  possible  to  initially  focus 
upon  the  invariants  that  are  found  when  looking  across  maps. 

The  first  step  in  the  process  of  generating  a  summary  concept  map 
involved  the  identification  of  the  key  concept  kernels  represented  in  each  of 
the  maps  (McFarren,  1987).  A  kernel  is  simply  a  cluster  of  concepts  that 
are  all  related  to  a  single  important  concept.  The  graphical  structure  of  the 
concept  maps,  as  discussed  above,  facilitates  the  identification  of  key  ideas 
within  a  subject  area  allowing  the  reader  of  the  map  to  immediately  identify 
global  characteristics  of  the  knowledge  domain  (Lambiotte,  et.  al.,  1989). 
Recognition  of  the  domain  characteristics  can  then  guide  the  map  reader 
during  the  extraction  of  detailed  meaning,  and  facilitate  the  identification 
of  the  concept  clusters  within  the  concept  map.  The  key  concept  within  the 
cluster  tended,  in  general,  to  be  the  parent  concept  in  the  cluster’s 
hierarchy  of  concepts.  Although  the  concept  clusters  were  principally 
identified  on  the  basis  of  there  being  “coherent  units”  within  the  larger 
map,  several  heuristics  were  used  to  aid  in  the  identification  of  the  key 
concept,  and  the  concept  clusters.  The  key  concepts  tended  to:  1)  have  a 
relatively  large  number  "f  connected  concepts,  resulting  from  the  fact  that 
they  had  been  discussec  A  considerable  length  by  the  pilot;  2)  be  the  parent 
concept  having  many  generations  of  related  concepts;  3)  be  a  concept  that 
was  invariant  across  maps;  and  4)  have  declarative  words  appear  as  a 
related  concept  (i.e.,  "this  is  important").  The  concept  clusters  are  then 
identified  as  including  the  set  of  relations  and  concepts  surrounding  the 
key  concept. 

Working  independently,  two  researchers  analyzed  the  individual  pilot 
maps  for  key  concepts  and  concept  clusters.  The  researcher’s  assessment 
of  the  concept  maps  and  the  identification  of  concept  clusters  resulted  in 
over  95%  agreement.  The  few  disagreements  were  resolved  by  reviewing 
the  audio  recording  of  the  mapping  session  in  order  to  clarify  specific 
contextual  issues. 

Once  the  concept  clusters  had  been  identified  in  each  of  the  individual 
pilot  maps,  they  were  literally  “cut  and  pasted”  to  the  summary  map.  Any 
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connections  that  were  broken  in  the  process  of  removing  the  cluster  from 
the  greater  context  of  the  concept  map  were  labeled  in  order  to  insure  that 
their  context  would  be  preserved.  The  isolated  clusters  were  checked  for 
internal  consistency,  and  then  compared  with  similar  clusters  that  had 
been  extracted  from  the  other  maps.  Both  the  invariants  and 
inconsistencies  among  the  concept  clusters  were  noted. 

Once  the  analysis  had  been  performed  on  the  entire  set  of  concept 
clusters,  they  were  then  aggregated  in  the  form  of  a  summary  map.  The 
summary  map  has  the  ability  to  represent  either  the  invariants  that  exist 
across  the  individual  concept  maps  or  the  full  breadth  of  knowledge  that 
has  been  captured  by  the  individual  concept  maps.  In  order  to  construct  the 
summary  map  that  captures  the  invariants,  each  cluster  would  be 
examined  and  only  those  found  to  have  common  concepts  and  relations 
would  be  included  within  the  summary  map.  In  order  to  represent  the  full 
breadth  of  knowledge  that  had  been  elicited,  the  clusters  would  be  examined 
and  the  full  set  produced  by  the  union  of  the  individual  concept  maps  would 
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be  included  in  the  summary  map.  The  form  that  the  summary  map  takes 
depends  largely  on  the  needs  of  the  knowledge  engineer  and  the  conditions 
under  which  the  knowledge  is  elicited.  The  utility  of  the  summary  map 
which  strictly  captures  the  invariants  depends  upon  the  size  and 
complexity  of  the  problem  domain,  and  the  number  of  available  domain 
experts.  When  dealing  with  extremely  large  and  complex  problem 
domains,  or  with  a  limited  number  of  domain  experts,  the  degree  of 
overlap,  or  invariants  found  across  maps  may  be  relatively  limited.  Under 
such  conditions,  it  may  be  more  desirable  to  produce  a  summary  map  that 
will  primarily  represent  the  union  of  the  individual  maps. 

Due  to  the  complexity  of  the  target  acquisition  phase  of  a  tactical  air-to- 
ground  mission,  and  the  relatively  small  number  of  domain  experts  that 
were  interviewed  during  the  concept  demonstration  phase  of  this  project, 
the  summary  map  that  was  constructed  represents  the  composite  of  all  the 
knowledge  elicited  during  the  interviews  with  the  eight  domain  experts. 
However,  in  addition  to  representing  a  breadth  of  knowledge,  a  great  degree 
of  invariance  in  the  pilot’s  conceptualization  of  this  mission  phase  was 
captured.  This  finding  may  run  counter  to  what  other  knowledge 
techniques  report  regarding  elicitation  of  pilot  knowledge.  Many  other 
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knowledge  acquisition  attempts  have  suggested  that  pilot  knowledge  cannot 
be  compared  as  a  consequence  of  many  insolvable  conflicts  and 
disagreements.  There  were  conflicts  but  our  assessment  found  that  the 
degree  of  invariance  completely  outweighed  the  extent  of  conflicts  between 
pilots’  knowledge.  Rather  than  extracting  conflicting  knowledge  about  the 
mission,  the  tendency  was  to  find  complementary  knowledge  from  each 
individual  pilot.  In  the  event  that  conflicts  are  found  during  the  summary 
process,  the  concept  map  provides  the  shared  media  wherein  the  conflict 
may  be  described  and  represented  for  future  discussion  and  resolution  (see 
Fraser,  Hipel,  Kilgore,  McNeese,  &  Snyder,  1989;  Hammond,  1987;  McNeese 
&  Snyder,  1987  for  additional  material  on  cognitive  conflict  resolution). 


2.11  Evaluation  of  the  Summary  Map 

Perhaps  the  most  effective  method  for  evaluating  the  summary  concept 
map  is  to  begin  by  assessing  the  global  pattern  of  concepts  as  they  appear  on 
the  map.  Starting  with  global  pattern  of  concepts,  an  overall  impression  of 
the  pilots’  knowledge  concerning  target  acquisition  can  be  obtained.  It  is 
possible  to  determine  the  concepts  that  the  pilots  consider  to  be  most 
important  and  how  these  concepts  relate  to  other  concepts  on  the  map.  This 
review  of  the  information  contained  in  the  concept  map  only  begins  to 
scratch  the  surface  of  what  is  there.  It  is  intended  to  give  the  reader  a 
sense  of  how  this  representation  of  domain  experts’  knowledge  can  be 
utilized  in  a  variety  of  ways. 

Upon  evaluation  of  the  summary  map,  one  of  the  most  important 
issues  relating  to  target  acquisition  is  the  preflight  planning.  The  whole 
success  of  the  mission  depends  upon  the  planning  that  occurs  before  the 
flight  leaves  the  ground.  The  pilots  have  indicated  that  what  they  do  during 
the  flight  is  (hopefully)  making  minor  adjustments  to  the  plan.  Given  the 
current  state  of  the  aircraft  and  its  systems,  the  pilots  tended  to  agree  that  a 
major  readjustment  would  be  tantamount  to  aborting  the  mission.  An 
examination  of  the  map  shown  in  Appendix  A  indicates  that  the  planning 
starts  with  the  target  and  works  backwards  to  the  IP,  taking  into 
consideration  such  things  as  target  characteristics,  weapon  type,  threat 
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potential,  predicted  weather  conditions,  terrain  features,  force  size,  and 
time  of  day.  These  key  concepts  are  related  to  decisions  concerning  route 
selection,  attack  parameters,  and  weapon  delivery  mode,  and  ultimately 
influence  the  selection  of  the  IP,  the  Action  Point,  and  the  WRP. 

An  evaluation  of  the  concept  map  should  also  consider  the  overall 
goals  and  objectives  of  the  technique.  The  discussion  below  is  not  an 
inclusive  analysis  of  the  concepts  and  interrelationships,  but  an  overview  of 
how  this  particular  concept  map  achieved  the  overall  goals.  A  few 
examples  are  used  to  illustrate  the  utility  of  the  map. 

Goals  of  concept  mapping: 

1.  As  noted  above,  one  of  the  goals  of  initial  concept  mapping 
sessions  was  to  identify  global  characteristics  of  the  knowledge 
domain  and  the  overall  complexity  of  that  domain. 

Furthermore,  the  objective  of  concept  mapping  is  to  facilitate 
the  identification  of  key  ideas  (concepts).  The  types  of 
information  found  in  the  summary  map  of  the  target 
acquisition  phase  of  an  air-to-ground  combat  mission  provide 
an  overview  of  the  knowledge  domain  from  preflight  planning 
to  IP  to  WRP.  This  first  cut  at  eliciting  knowledge  from  domain 
experts  provides  a  foundation  from  which  to  identify  major 
problem  areas,  areas  rich  for  further  information 
requirements  analysis  by  concept  mapping,  and  even 
information  about  helpful  improvements  from  the  pilot's  point 
of  view.  The  upper  right-hand  corner  of  the  summary  concept 
map  illustrates  an  example  of  pilots'  suggestions  for  "possible 
improvements"  to  make  target  acquisition  easier.  These 
improvement  concepts  are  related  to  "sensor"  sensitivity, 

"head-up  displays",  and  "display  content"  requirements. 

2.  Another  goal  of  concept  mapping  is  to  provide  an 
understanding  of  the  nuances  of  the  knowledge  domain.  The 
summary  map  for  the  target  acquisition  task  illustrates  the 
potential  for  variation  in  the  combat  environment.  It  was 
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noted,  for  example,  on  the  section  of  the  map  showing  the 
concepts  and  relations  for  preflight  planning/planning 
considerations  that  "time-of-day",  "weather",  "target 
characteristics",  and  prediction  of  "potential  problems", 
among  many  other  variations,  impact  the  mission  plan  and 
tactics  plans.  In  addition,  the  concept  map  provided 
information  regarding  the  dynamic  environment  by  the 
concepts  of  "compensating  for  differences",  "modifying  plans" 
and  "timing  problems"  that  may  occur. 

3.  The  identification  of  problem  areas  as  viewed  by  domain 
experts  is  another  goal  of  knowledge  elicitation  by  concept 
mapping.  There  are  examples  of  two  types  of  problem  areas  in 
the  summary  map  for  target  acquisition.  One  of  these  areas 
relates  to  problems  that  occur  unpredictably  during  the  course 
of  a  mission.  These  problems  are  addressed  under  the 
"preflight  planning"  concept  and  are  identified  as  "problems" 
nodes.  Some  ot  these  problems  refer  to  "threat  encounter", 
"changes  in  force  size"  or  possibly  problems  due  to  "weather". 

Another  source  of  problems  that  were  identified  on  the 
concept  map  relates  to  the  "aircraft  systems"  concept  and  the 
display  characteristics  of  the  "HUD".  For  example,  the  field  of 
view  is  "too  narrow",  it  is  "like  looking  through  a  straw".  In 
noting  the  cluster  of  concepts  related  to  possible  improvements, 
the  fact  that  unnecessary  data  may  currently  be  cluttering  the 
display  is  indicated  by  the  concepts  of  "display  content"  should 
be  "void  of  unnecessary  data".  Sensor  sensitivity,  "see  through 
smoke  and  water”  and  incompatibility  of  "mental  model"  and 
display  mode  are  also  indications  of  possible  problems  that 
should  be  addressed  in  further  investigations  by  the  knowledge 
engineers  and  the  design  team,  and  serve  as  indications  of 
human  factor  problems  elicited  from  the  pilot. 


4.  Identification  of  information  requirements  is  the  major 
reason  for  acquiring  expert  knowledge  by  concept  mapping.  To 
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the  extent  that  the  domain  experts  are  capable  of  identifying 
the  information  that  they  use  to  make  the  various  decisions 
relating  to  their  mission,  it  becomes  possible  to  use  the  concept 
map  to  assist  in  the  design  of  a  decision  support  system  and 
the  design  of  the  pilot-vehicle  interface.  There  are  several 
concepts  relating  to  information  requirements.  For  example, 
some  pilots  indicated  that  communications  between 
themselves  and  their  backseater  needed  to  be  kept  explicit  (to 
prevent  a  loss  of  situation  awareness).  It  implies  by  analogy 
that  the  communications  between  the  pilot  and  an  intelligent 
PA  should  be  similarly  explicit.  In  fact,  as  the  concept  map 
reflects,  when  the  pilots  were  queried  about  how  they  would 
choose  to  interact  with  an  intelligent  system,  they  could  not 
imagine  feeling  comfortable  or  trusting  a  system  that 
attempted  to  infer  their  intentions  on  the  basis  of  their  actions. 
The  need  for  explicit  communications  between  an  intelligent 
system  and  themselves  (as  pilots)  was  something  that  the 
pilots  emphasized  rather  strongly.  They  wanted  to  be  able  to 
issue  commands  to  the  Pilot’s  Associate,  and  receive  feedback 
from  the  system  in  order  to  verify  that  their  request  had  been 
accurately  executed. 

The  map  is  also  a  source  of  many  other  concepts 
regarding  information  about  the  pilots’  information 
requirements.  At  the  more  global  level,  a  reading  of  the 
concept  map  indicates  that  the  pilots  need  to  have  information 
specifying  the  spatial  and  temporal  relationships  between 
themselves  and  the  target,  between  themselves  and  the 
ground,  and  between  themselves  and  potential  threats.  In 
addition  the  pilots  indicate  that  they  want  to  keep  their  “heads 
up”,  and  look  out  of  the  cockpit  during  me  entire  target 
acquisition  phase  of  the  mission. 

5.  The  final  goal,  to  develop  a  major  communication  device 
in  which  domain  knowledge  is  transmitted  from  user  to 
designer,  was  discussed  earlier.  The  aforementioned  work  by 
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Nosek  and  Roth  on  the  success  of  semantic  networks  as  a 
knowledge  representation  technique  for  AI  indicates  the 
usefulness  of  this  representation  scheme  from  the  designer's 
point  of  view.  That  this  type  of  representation  scheme  can  be 
elicited  directly  from  the  domain  expert  has  been 
demonstrated. 


2.11.1  Information  Gleaned  from  the  Summary  Concept  Man 

First  and  foremost,  a  global  description  of  the  target  acquisition  task 
was  achieved;  and  that  is,  the  goal  of  preflight  planning  is  to  reduce  the 
unknowns  and  the  decisions  that  the  pilot  makes  during  the  mission. 
Furthermore,  the  focus  of  pilot  activities  during  flight  is  in  testing  these 
preflight  plans  against  the  reality  of  the  dynamic  unpredictable 
environment  of  air-to-ground  combat  missions.  In  other  words,  reality 
testing  is  an  implicit  global  concern  found  by  concept  map  analysis. 

2.11.1.1  Kev  Decision  Points  The  domain  knowledge  represented  on 
the  concept  map  takes  many  different  forms.  It  depends  on  the  goals  of  the 
knowledge  engineer,  and  the  design  goals,  as  to  just  how  this  knowledge  is 
utilized.  Information  requirements  can  be  derived  based  on  time-line 
characteristics,  for  example.  At  critical  decision  points  during  the 
mission,  concepts  and  their  relationships  to  other  concepts  can  be 
examined.  There  were  six  major  decision  points  for  the  target  acquisition 
phase  of  the  mission  illustrated  on  the  concept  map:  IP,  Action  point,  Pull¬ 
down,  Roll-in,  Track  point,  and  Weapon  release  point.  If  we  take  the  Pull¬ 
down  point  as  an  exemplar  decision  point,  then  it  can  be  seen  that  action 
adjustments  are  made  and  that  the  actions  require  specific  types  of 
information  (e.g,  current  altitude,  planned  pull-down  altitude,  parameters 
pertaining  to  deviations  in  planned  attack  geometry,  etc.). 

2.11.1.2  Identification  of  Kev  Concepts  One  of  the  major  goals  of 
concept  mapping  was  to  identify  key  concepts  of  the  knowledge  domain. 
Examination  of  the  summary  map  revealed,  as  was  noted  previously,  that 
"preflight  planning"  drives  the  target  acquisition  process.  Preflight 
planning  and  subordinate  concepts  take  up  mere  than  one- half  the  map 
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and  has  relationships  with  all  other  concepts  on  the  map.  This  indicates 
that  the  "preflight  planning"  concept  has  a  role  as  an  executive  concept. 
Further  examination  of  the  map  will  indicate  that  during  the  actual 
segment  of  flight,  all  activities  revolve  around  the  testing  and  "reaction  and 
readjustment"  to  accommodate  the  preflight  plan. 

Other  concepts  identified,  using  the  criteria  of  hierarchy  and  number 
of  subordinate  concepts  to  choose  these  key  concepts,  were: 

Communication:  has  a  subordinate  concept  noting  that 
target  acquisition  depends  on  "effective  communication" 
between  "backseater"  or  "Pilot's  Associate".  Some  of  the 
concepts  dealing  with  communication  provide  information 
regarding  the  pilot's  concept  of  effective  communication, 
including,  for  example,  "gives  information"  (as  opposed  to 
data),  "identifies  problems",  "does  not  control",  and  "gives 
direct  commentary". 

Visual  Acquisition:  concepts  that  link  with  "visual 
acquisition"  such  as  "IP",  "pop-up",  and  the  superordinate 
concept  of  "target  acquisition"  and  cpncepts  that  impact  visual 
acquisition  provide  the  type  of  information  that  indicates  where 
the  pilot's  visual  attention  must  be  focused.  That  is,  incoming 
information  to  the  pilot  suffers  those  constraints. 

Furthermore,  the  concept  of  "visual  acquisition"  to  acquire 
target  has  multiple  connections  to  the  "preflight-planning" 
concept.  For  example,  use  of  the  strategy  of  "big  to  small" 
relies  on  characteristics  of  "most  prominent  features" 
identified  in  "aerial  photos",  a  major  concept  of  "preflight 
planning".  Pilots  also  indicated  that  an  adjustment  to  the 
initially  presented  maps  during  preflight  planning  must  be 
accomplished  via  mental  rotation  during  the  course  of  visual 
acquisition  and  that  one  of  their  statements  of  need  for  visual 
assessment  technology  would  be  an  ‘assistant’  which  would 
help  them  ‘calibrate’  this  mental  rotation  in  flight. 
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Aircraft  systems:  the  aircraft  systems  represented  on  the 
concept  map  consists  of  "HUD",  "INS"  and  "RWR".  Those 
systems  provide  data/information  and  present  a  direct  impact 
on  the  pilot.  The  "HUD"  is  represented  in  two  places  on  the 
concept  map,  once  under  "possible  improvements"  and  once  as 
a  key  concept  directly  related  to  "aircraft  subsystems".  Some 
sample  concepts  regarding  "HUD"  requirements  note  that  the 
"HUD"  should  "display"  and  "monitor  by  exception".  This 
concept  addresses  the  concern  that  status  data  not  of 
immediate  concern  should  be  suppressed  in  some 
circumstances. 

2.11.1.3  Problems  and  Problem  Solutions  Additional  information 
found  on  the  concept  maps  refers  to  the  types  of  problems  or  tasks  the  pilot 
faces  during  the  target  acquisition  phase  of  the  mission.  We  noted  earlier 
that  much  of  the  summary  concept  map  consisted  of  discussion  about 
mission  concepts  that  are  addressed  in  the  preflight  planning  stage.  Then 
it  was  noted  that  the  actual  mission  consisted  of  attempting  to  accomplish 
the  mission  according  to  plan.  A  time-line  representation  of  mission 
activities  revealed  concepts  that  addressed  this  constant  testing  of  the 
mission  plan  against  the  reality  of  the  dynamic  mission  environment. 

The  concepts  that  were  revealed  addressed  the  types  of  tasks  and 
decisions  that  were  actually  being  effected.  Concepts  such  as 
"compensating  for  differences",  "modifying  plans",  "making  last  minute 
corrections",  "planning"  and  "evaluation"  all  address  the  dimension  of 
task  type  that  in  turn  speaks  to  the  cognitive  activities  and  pilot  resources 
required  to  accomplish  the  combat  mission.  Pilot  tasks,  therefore,  can  be 
classified  in  terms  of  management,  monitoring,  prediction,  classification, 
and  interpretation  requirements. 

These  concepts  reveal  yet  another  concern  for  information 
requirements  analysis.  The  type  and  mode  of  presentation  of  required 
information  must  take  into  account  pilot  resource  requirements.  The 
concept  map  points  out  the  need  for  investigation  into  those  questions  and, 
in  addition,  pinpoints  specific  concepts  related  to  these  requirements.  In 
other  words,  the  PA  system  decisions  can  address  these  concerns  based  on 
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the  concept  map. 

2.11.1.4  Information  Requirements  Information  requirements  are 
not  represented  under  a  concept  of  "information  requirements"  but,  instead 
are  embedded  throughout  the  summary  map.  Each  major  concept  has 
associated  with  it  information  that  is  presented,  needed  or  available  and 
therefore,  addressed  by  the  knowledge  engineer  and  design  team.  For 
example,  "navigation  data",  "target  data",  "relationships  of  self  to  route"  or 
"ground",  "threat  position",  etc.  are  represented  as  concepts  under 
"aircraft  systems".  "Visual  acquisition"  relies  on  information  preplanned 
and  out-of-the-window.  The  information  represented  in  the  concept  map  is 
interconnected  and  related  to  several  concepts  in  a  myriad  of  ways. 

As  noted  earlier,  the  evaluation  of  the  summary  map  cannot  be 
comprehensive  at  this  time.  We  merely  wish  to  provide  a  focus  from  which 
to  view  the  summary  concept  map.  This  initial  analysis  of  the  pilot¬ 
generated  concept  map  provides  a  systematic  method,  a  front-end  analysis, 
so  to  speak,  to  acquire  a  global  view  of  the  characteristics,  key  concepts  and 
information  requirements  from  the  pilot's  point  of  view. 


2.12  Observations 

Concept  mapping  has  proved  to  be  an  effective  method  for  capturing 
the  domain  expert’s  conceptualization  of  the  problem  domain.  As  the 
preceding  section  demonstrated,  the  concept  mapping  technique  has  been 
shown  to  be  an  effective  method  for  identifying  the  user  requirements.  The 
concept  mapping  technique  offered  the  opportunity  to  access  two 
interrelated  processes  pertaining  to  the  pilot’s  conceptualization  of  the 
mission.  The  technique  offered  the  opportunity  to  acquire  the  ‘information 
heeded’  and  ‘information  remembered’  depending  upon  the  level  of 
intrusion  (i.e.,  probing)  that  occurred  during  the  elicitation  process.  The 
information  heeded  was  elicited  as  the  pilot  was  allowed  to  “think  aloud”, 
wherein  there  was  minimal  amount  of  intrusion  by  the  interviewer.  The 
pilot  was  afforded  the  opportunity  to  spontaneously  access  information  that 
he  or  she  deems  related  to  the  particular  phase  of  the  mission  that  was 
under  consideration  without  the  imposition  of  the  knowledge  engineer’s 
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point  of  view  (i.e.,  uninformed  access).  The  concept  mapping  technique 
also  offered  the  opportunity  to  elicit  the  ‘information  remembered’  as  the 
amount  of  intrusion  was  increased.  As  the  pilot  responded  to  a  given  probe, 
he  or  she  was  encouraged  to  access  specific  information  (i.e.,  informed 
access).  This  information  proved  valuable  in  that  it  frequently  led  to  the 
generation  of  a  more  highly  differentiated  concept  map. 

The  issue  of  validation  is  a  particularly  important  one,  as  well  as  being 
one  that  knowledge  acquisition  techniques  have  traditionally  had  difficulty 
addressing.  Following  Nosek  &  Roth’s  (1990)  metaphor  which  equates 
knowledge  acquisition  techniques  with  conduits  of  information  that  may 
have  impurities  or  filters  of  some  kind,  it  is  important  to  insure  that  the 
transference  of  knowledge  has  been  complete  and  without  the  inclusion  or 
systematic  exclusion  of  impurities.  To  this  date,  the  domain  expert 
validation  remains  the  single  most  highly  regarded  measure  of  the  general 
success  of  a  knowledge  acquisition  technique.  In  other  words,  the  domain 
experts  are  usually  asked  to  evaluate  the  completeness  and  accuracy  of  the 
knowledge  representation  that  is  being  used  to  capture  the  domain 
knowledge.  In  this  regard,  concept  mapping  has  received  extremely 
positive  reviews. 

The  process  of  concept  mapping  was  unequivocally  described  by  the 
pilots  as  a  very  effective  way  for  them  to  express  their  understanding  of  the 
problem  domain.  Unlike  the  more  traditional,  and  more  structured,  verbal 
protocol  techniques,  they  indicated  that  concept  mapping  had  permitted 
them  to  explore  the  numerous  contingencies  associated  with  the  problem 
domain.  They  felt  that  their  discourse  had  not  been  limited  by  the  nature  of 
the  knowledge  acquisition  tool  to  a  single  scenario  as  would  typically  occur 
with  the  use  of  less  open  ended  verbal  protocol  techniques.  It  was  generally 
felt  that  the  concept  mapping  technique  genuinely  enhances  the 
presentation  of  their  knowledge.  Several  of  the  pilots,  in  addition  to  being 
regarded  as  domain  experts,  were  also  individuals  that  have  had  a 
significant  amount  of  exposure  to  a  wide  variety  of  knowledge  acquisition 
techniques.  Their  impressions  of  the  concept  mapping  technique  were 
especially  valued,  and  one  of  these  pilots  stated  that,  “the  concept  mapping 
technique  was  the  best  knowledge  acquisition  technique  that  I  have 
encountered,  and  I  have  encountered  quite  few.  It  lets  me  be  sure  that 
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you  [the  knowledge  engineer]  are  correctly  understanding  what  I  am 
saying  and  not  simply  hearing  what  you  want  to  hear.” 

In  addition  to  the  many  positive  attributes  that  the  concept  mapping 
technique  brings  to  the  field  of  knowledge  acquisition,  the  knowledge 
engineer  considering  the  use  of  this  technique  also  needs  to  be  aware  of  the 
potential  weaknesses  that  were  encountered.  First,  from  a  human  factors’ 
perspective,  it  appears  that  the  complexity  of  the  representation  may 
possess  problems  in  terms  of  comprehension.  As  stated  above,  one  of  the 
advantages  in  using  the  concept  mapping  technique  was  said  to  be  the 
graphical  quality  of  the  representation  which  could  enable  a  viewer  of  the 
map  to  quickly  glean  the  pilot’s  conceptualization  of  the  problem  domain. 
However,  as  the  complexity  of  the  problem  domain  increases,  there  is  a 
corresponding  increase  in  the  complexity  of  the  representation  of  that 
domain,  as  reflected  in  the  concept  map.  There  may  reach  a  point  at  which 
the  concept  map  has  become  too  complex,  with  too  many  concepts  and 
relations  for  an  individual  to  easily  comprehend.  The  summary  map  (see 
Appendix  A)  may  be  approaching,  or  perhaps  already  exceeding,  this 
critical  boundary. 

Because  the  complexity  of  the  map  is  being  driven  by  the  complexity  of 
the  domain,  any  attempt  to  simplify  the  map  must  be  carefully  considered. 
Simply  parsing  the  map  into  subsections  on  the  basis  of  related  concept 
clusters  would  offer  an  intuitive  solution  to  the  complexity  problem. 
Unfortunately,  the  solution  would  negate  one  of  the  principle  advantages 
inherent  in  this  type  of  representation,  namely  the  ability  to  see  all  the 
relations  between  the  numerous  concepts  portrayed  in  a  single  integrated 
format.  Another  intuitive,  and  potentially  more  satisfactory  solution  to  the 
complexity  issue  would  involve  parsing  the  concept  map  according  to  its 
hierarchical  organization.  In  the  same  way  that  a  highway  map  of  the 
United  States  only  depicts  the  principle  arteries  and  those  of  special 
significance  (i.e. ,  scenic  routes,  and  small  but  vital  routes),  the  concept 
map  could  at  one  level  only  represent  the  key  concepts.  This  high-level  map 
would  preserve  the  global  structure  of  the  network  of  concepts  and  enable 
the  viewer  to  appreciate  the  relationships  that  exist  between  the  key 
concepts.  Important  aspects  of  the  heterarchical  structure  could  also  be 
preserved  by  including  relevant  subordinate  concepts.  When  more  detailed 
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information  is  required,  the  key  concepts  could  be  used  to  access  the 
underlying  clusters,  in  the  same  way  that  the  global  structure  of  the 
highway  maps  are  used  to  reference  the  embedded  street  maps  that  provide 
a  finer  grain  description. 

Another  potentially  serious  problem  with  the  concept  mapping 
technique  of  which  the  knowledge  engineer  needs  to  be  cognizant  is  the 
potential  loss  of  the  interactive  aspects  of  the  technique.  The  interactive 
quality  of  concept  mapping  is  one  of  the  technique’s  unique  attributes  that 
enables  the  knowledge  engineer  to  have  confidence  in  the  veridicality  of  the 
knowledge  representation.  It  is  important  for  the  domain  expert  to  be 
continuously  attending  to  the  concept  map  that  is  being  drawn  in  front  of 
him  or  her  to  be  correcting  any  misrepresentations  of  the  domain 
knowledge.  In  several  instances,  pilots  were  simply  relaying  their 
information  to  the  knowledge  engineer  without  simultaneously  looking  at 
what  was  being  drawn  on  the  map.  Fortunately,  the  solution  to  this 
potential  difficultly  merely  requires  the  knowledge  engineer  to  occasionally 
pause  in  the  interview  and  question  the  domain  expert  about  the  accuracy 
of  the  map  (i.e.,  “Is  this  what  you  mean?”  or  “Does  this  correctly  show  how 
these  concepts  are  related?”).  These  types  of  probes  can  also  be  considered 
as  relatively  non-obtrusive,  and  therefore  unlikely  to  change  the  focus  of  the 
interview  to  one  of  informed  access,  because  they  do  not  specifically  alter 
the  focus  of  the  expert’s  attention  with  regard  to  the  knowledge  domain. 

The  result  is  simply  to  insure  that  the  concepts  the  domain  expert  considers 
to  be  important  get  accurately  represented  in  the  concept  map. 
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3  STRUCTURED  TASK  DECOMPOSITION:  AN  ANALYSTS  VIEW  OF 
THE  MISSION 

3.0  The  IDEFq  Technique 

Numerous  techniques  have  been  designed  to  aid  in  the  understanding 
of  complex  human-machine  systems  and  processes,  and  a  substantial 
number  of  these  have  been  utilized  at  the  Armstrong  Aerospace  Medical 
Research  Laboratory  with  varying  degrees  of  success  (see  Table  3-1).  Of 
these  techniques,  the  Integrated  Computer-Aided  Manufacturing 
Definition0  (IDEF0)  approach  was  deemed  the  most  appropriate  for 
representing  the  analyst’s  perspective  of  the  user  requirements  (i.  e.,  the 
mission  description). 

In  order  to  generate  a  functional  task  decomposition  of  the  tactical  air- 
to-ground  mission,  the  IDEF0  technique  was  employed.  This  technique 
provides  a  structure  and  precise  semantics  that  are  used  in  order  to  control 
the  description  of  a  given  system  (Marca  &  McGowan,  1988).  The  activities 
or  functions  of  the  system  are  given  a  semantic  label.  The  activities  of  the 
system  are  then  depicted  as  boxes  within  a  hierarchical  representation 
scheme.  By  convention,  each  side  of  the  IDEFq  box  has  a  specific  meaning 
which  constrains  the  type  of  entity,  information  or  data  associated  with  the 
activity,  and  the  nature  of  the  interaction,  or  influence  that  a  particular 
entity  has  upon  the  activity.  Specifically,  the  left  side  of  the  box  is  reserved 
for  inputs,  the  top  of  the  box  for  controls,  the  right  side  for  outputs,  and  the 
bottom  side  of  the  box  is  reserved  for  mechanisms  (see  Figure  3-1). 

The  entities  and  activities  of  human-machine  systems  or  processes  are 
described  using  IDEF0  boxes  and  arrows.  The  arrows  represent  virtually 
anything  that  has  the  capacity  to  influence  the  activity,  or  results  from  the 
activity  that  is  depicted  by  the  box,  including  such  things  as:  information, 
data,  plans,  technical  orders,  resources,  or  the  product  of  the  activity. 

There  are  four  classes  of  arrows  corresponding  to  the  four  sides  of  the 
IDEFq  box.  The  input  arrows  (those  connecting  to  the  left,  or  input  side  of 
the  IDEFq  box)  are  defined  as  representing  those  things  that  are  used, 
and/or  transformed  by  the  activity.  The  control  arrows  represent  the  types 
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TABLE  3-1 


System  Modeling  and  Knowledge  Representation  Techniques 
Previously  Used  at  AAMRL 


1.  Outlines 

Used  for:  task  analysis,  goal  hierarchies,  work  breakdown 
Primary  data  structures:  text 
Representations  of:  top-down  hierarchy 
References:  Various 


2.  Block  diagrams 

Used  for:  functional  diagrams,  organizational  diagrams 
Primary  data  structures:  blocks,  directed  connections, 
explicit  levels  (if  hierarchical) 

Representations  of:  functional  elements,  organizational 
elements,  top-down  hierarchy,  connectivity, 
associations,  feedback  loops,  command  levels, 
processes 

References:  Various 


3.  Thsk/Decision  Trees 

Used  for:  task  analysis,  decision  analysis,  work  breakdown 
Primary  data  structures:  blocks,  undirected  connections, 
implicit  levels  (if  hierarchical) 

Representations  of:  top-down  hierarchy,  logic  links, 
inferences 

References:  Awad(1979);  Wohl  and  Tenney  (1987) 


4.  Program  Evaluation  and  Review  Technique/Critical  Path 

Methods  (PERT/CPM) 

Used  for:  task  analysis,  resource  analysis 
Primary  data  structures:  task  identifiers,  directed 
connections,  paths 

Representations  of:  bottom-up  tasks/events,  resources, 
performance  specifications/timing 
References:  Awad  (1979);  Wohl  and  Tenney  (1987) 
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Table  3-1  continued 


5.  State  Transition  Diagrams/Markov  State  Models 

Used  for:  analysis  of  state  transitions  within  a  system  or 
process 

Primary  data  structures:  system  state  identifiers,  directed 
connections,  implicit  transition  conditions, 
probabilities 

Representations  of:  system  state  transitions 

References:  Shaw  (1987) 


6.  Flow  Charts 

Used  for:  information  flow  analysis 
Primary  data  structures:  function  blocks,  directed 
connections,  decision  points,  sources/sinks,  logic 
elements 

Representations  of:  functions,  decisions,  control  flow, 
input/output,  terminal  points  (entrance/exit) 
References:  Wohl  &  Tenney  (1987) 


7.  Data  Flow  Diagrams  (DFD) 

Used  for:  information  flow  analysis 
Primary  data  structures:  task/process  circles,  directed 
connections,  source/sink  blocks,  information/object 
storage  points  (time  delays) 

Representations  of:  tasks,  processes,  procedures,  data 
stores,  external  organizations,  flow  of  information, 
flow  of  materials/objects,  input/output,  flow  timing 
References:  DeMarco  (1978);  Awad  (1979);  Wohl  &Tenney 
(1987);  Martin  (1989) 
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Table  3-1  continued 


8.  Structured  Analysis  and  Design  Technique/Integrated 

Computer  Aided  Manufacturing  (ICAM)  Definition^) 

(SADT/IDEFq)  diagrams 

Used  for:  hierarchical  functional  analysis  of 
activities, work  breakdown 

Primary  data  structures:  activity  blocks,  directed 

connections  (inputs,  outputs,  mechanisms,  controls), 
explicit  levels  (of  hierarchy),  implicit  priority  levels  (of 
activities) 

Representations  of:  top-down  hierarchy,  tasks,  activities, 
processes,  functions,  input/output  (status), 
consumable/nonconsumables  resources,  controls, 
priority 

References:  Ross  &  Schoman  (1977);  Hoyland,  Evers,  & 
Snyder  (1985);  Wohl  &  Tenney  (1987) 


9.  Structured  Analysis  of  an  Integrated  Network  of  Tasks 

(SAINT) 

Used  for:  structured,  event-driven  simulation  of  task 
networks 

Primary  data  structures:  task  nodes,  information 

attributes,  resource  attributes,  system  attributes,  task 
timing,  branching  criteria,  resource  clearing,  state 
variables,  state  variable  regulators,  task  monitors, 
moderator  functions,  probability  distributions,  network 
modifications,  output  options 
Representations  of:  task  networks,  complex  systems 
References:  Wortman,  et  al.  (1978);  Snyder  &  McNeese 
(1987) 


Table  3-1  continued 


10.  Petri  Nets/Colored  Petri  Nets  (CPN) 

Used  for:  dynamic  queuing  network  models 

Primary  data  structures:  tokens,  places,  transitions,  arcs, 
rules,  attributes,  ordered  sequence  of  transitions/ 
places 

Representations  of:  forces,  messages,  data,  intel, 
personnel,  equipment,  facilities,  functions, 
procedures,  processes,  communication  links, 
direction,  coordination,  control,  missions/complex 
systems 

References:  Wohl  &  Tenney  (1987);  Meta  Software 
Corporation  (1989) 
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Figure  3-1  IDEF0  Syntax 
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of  things  that  constrain,  or  control  activities,  such  as  plans,  technical 
orders,  or  rules  of  engagement.  The  output  arrows  represent  the  things 
into  which  the  inputs  have  been  transformed,  or  the  product  of  the  activity. 
For  example,  if  the  input  was  a  mission  status,  the  output  might  be  the 
change  in  that  status  resulting  from  a  performance  of  the  identified 
activity.  Lastly,  the  mechanism  arrows  represent  the  mechanism  or 
resources  that  are  being  utilized  by  the  system  in  order  to  accomplish  the 
activity  that  has  been  depicted  by  the  IDEF0box. 

The  influence  that  different  activities  have  on  each  other  take  many 
forms,  and  the  IDEF0  technique  has  been  designed  to  describe  and 
represent  the  various  types  of  interactions  that  are  likely  to  occur.  The 
arrows  are  used  to  represent  the  relationships  that  exist  between  the 
activity  boxes,  or  in  other  words  how  the  activities  influence  each  other.  For 
example  the  output  of  one  activity  may  be  required  as  an  input  to  another 
activity,  or  the  output  of  one  activity  serving  as  control  information  that 
governs  the  performance  of  another  activity.  In  addition,  although  perhaps 
rarely,  the  output  one  activity  has  produced  may  become  a  resource  that  is 
employed  by  another  activity.  The  IDEF0  technique  is  also  capable  of 
depicting  various  feedback  loops  or  iterations  in  the  execution  of  one  activity 
that  are  being  influenced  by  the  output  of  a  separate  activity. 

Each  function  of  the  system  or  process  is  decomposed  into  a  series  of 
interconnected  IDEF0  diagrams  that  graphically  depict  the  characteristics 
of  the  function  and  relationships  between  functions  (see  Figure  3-2).  The 
overall  system  or  process  is  represented  as  a  single  activity  in  a  top-level 
(Level  0)  diagram.  A  key  characteristic  of  the  IDEF0  model  is  that  it  is 
hierarchical  in  nature.  Each  layer  is  an  elaboration  of  the  layer  (function) 
above  it.  Each  child  inherits  the  attributes  of  its  parents  and  passes  on  its 
own  attributes  to  its  children.  Thus,  the  architecture  supports  the 
establishment  of  a  top-down  design  which  can  be  used  to  establish  a 
prioritization  of  constraints  and  controls  based  on  hierarchy.  The 
functional  requirements  are  developed  by  determining  the  specific  tasks, 
decisions,  information  requirements  and  pilot  actions  occurring  during  the 
performance  of  the  mission. 

The  activities  or  functions  that  are  accomplished  by  the  system  can  be 
broken  into  sub-level  layers  by  means  of  a  functional  task  decomposition. 
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In  this  decomposition,  each  activity  is  defined  as  a  function  or  task,  and  is 
represented  with  symbols  that  graphically  depict  the  interactions  and 
relationships  that  exist  with  other  tasks  or  functions.  Each  function  is 
dependent  upon  various  inputs,  provides  particular  outputs,  operates  via 
various  mechanisms  and  performs  its  activity  according  to  specified 
restrictions,  guidelines,  controls  and  constraints.  For  every  activity 
(function  or  task)  key  decision  elements  can  be  identified  and  precise 
descriptions  of  the  network  of  functions,  relations,  and  corresponding 
parameters  can  be  determined. 

In  addition  to  decomposing  the  activities  that  are  performed  by  the 
system,  the  IDEF0  technique  also  allows  for  both  the  decomposition  and 
integration  of  the  various  entities  that  influence  the  system’s  activities.  The 
arrows  in  an  IDEFq  diagram  typically  represent  a  collection  of  things 
having  multiple  sources  and  multiple  destinations,  and  as  a  consequence, 
the  arrows  can  be  represented  as  branching  apart  and  joining  together  in 
complex  ways. 


3.1  An  IDEFq  Description  of  the  Tactical  Air-to-Ground  Mission 

The  construction  of  an  IDEFq  description  generally  requires  extensive 
domain  knowledge,  and  the  process  of  using  IDEFq,  as  a  consequence, 
generally  begins  with  the  collection  of  that  knowledge  (Marca  &  McGowan, 
1988).  For  the  present  context,  the  information  was  provided  by  one  of  the 
authors,  John  Duncan,  who  has  extensive  experience  in  the  area  of  fighter 
pilot  training,  simulation,  modeling,  and  avionics  systems  design.  This 
knowledge  and  experience  with  fighter  aircraft  operational  requirements 
and  avionics  engineering  design  is  regarded  as  crucial  for  the  performance 
of  the  task  analysis,  requirements  definition,  and  information  modeling. 

Because  IDEFq  is  only  capable  of  representing  a  single  purpose, 
viewpoint,  and  context,  it  is  important  that  these  attributes  be  clearly 
defined  before  constructing  the  diagrams.  The  decomposition  of  the 
activities  being  represented  in  an  IDEFq  model  typically  proceeds  until  the 
analyst  has  either  exhausted  the  extent  of  his  or  her  knowledge  of  the 
system,  or  until  the  model  has  satisfied  its  stated  purpose. 
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Figure  3-2  IDEF0  Hierarchical  Decomposition 


The  IDEFq  representation  of  the  tactical  air-to-ground  mission  is 
provided  in  Appendix  B.  The  viewpoint  was  assumed  to  be  that  of  the 
pilot’s.8  The  context  for  this  IDEFq  model  was  identified  as  being  a  tactical 
air-to-ground  mission  as  performed  by  an  F-16  type  aircraft  with 
Continuously  Computed  Impact  Point  (CCIP)  and  Continuously  Computed 
Release  Point  (CCRP)  weapon  delivery  modes,  and  aircraft  fire  control, 
radar,  and  related  weapon  systems  having  comparable  accuracy  and 
capabilities.  An  emphasis  was  placed  on  the  associated  mission  functions 
occurring  during  the  ingress  to  the  target  IP  phase,  the  target  attack 
phase,  and  the  egress  phase.  This  representation  depicts  the  activities  that 
the  pilot  must  accomplish  in  order  to  successfully  complete  the  mission.  A 
determination  of  the  decisions,  tasks,  information  requirements,  and  pilot 
actions  that  occur  during  the  performance  of  a  tactical  air-to-ground 
mission  was  accomplished  during  the  decomposition  process.  Task 
performance  was  modeled  down  to  the  switch  activation  level  in  order  to 
demonstrate  the  model’s  capacity  of  providing  a  means  for  performing 
analyses  on  such  things  as  workload,  computer  control  of  switches,  panel 
layout,  control  mechanisms,  and  task/function  allocation. 

In  addition  to  an  IDEF0  model  of  a  tactical  air-to-ground  mission,  an 
IDEF0  Data  Dictionary  was  created  to  provide  definitions  for  each  function 
within  the  IDEF0  Tactical  Air-Ground  Mission  decomposition  (see 
Appendix  C).  Unique  Inputs,  Outputs,  £ontrols/£onstraints,  ancj 
performance  Mechanisms  were  described  and  descriptions  of  pertinent 
parameters  are  generated  for  those  functions  requiring  elaboration. 


3.2  Observations 


3.2.1  Strengths.  A  primary  characteristic  and  strength  of  the  IDEFq 
technique  was  its  incorporation  of  trait  inheritance  through  a  hierarchy  of 
functional  decompositions  and  relationships.  For  the  purposes  of  the 
tactical  air-to-ground  mission,  there  were  several  continuous  concerns 

*  Note,  that  although  the  viewpoint  is  said  to  be  that  of  the  pilot’s,  it  does  not  serve  to  alter 
the  perspective  that  is  represented  using  this  technique.  The  analyst,  in  this  case,  is 
attempting  to  assume  the  pilot’s  viewpoint  during  the  performance  of  a  tactical  air-to- 
ground  mission,  and  thus,  represents  the  analyst’s  perspective  of  the  user  requirements. 
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involved  with  the  operation  of  the  aircraft  and  success  of  the  mission. 

These  items  are  persistent  in  nature,  and  at  various  times  took  precedence 
over  all  other  actions.  For  instance,  of  prime  concern  at  all  times  are  those 
conditions  that  are  immediately  life-threatening,  or  potentially  fatal  to  the 
pilot,  such  as  catastrophic  aircraft  failure,  or  impending  threat 
encounters.  In  those  cases,  aircraft/pilot  survival  actions  take  immediate 
precedence  over  ongoing  tasks  -  the  pilot  must  perform  those  actions  that 
will  alleviate  the  circumstances  that  are  life  threatening  at  the  expense  of 
the  current  tasks  and  possibly  mission  goals.  The  structure  of  IDEF0  easily 
allowed  high-level  traits  to  be  represented,  while  the  analysis  of  other 
functions  were  traced  to  lower  levels. 

When  the  IDEFq  was  expanded  to  the  lowest  levels,  the  systems 
management  actions  and  discrete  pilot  actions,  including  switch 
operations  of  particular  equipment,  were  explicitly  shown.  By  employing 
this  method  of  increasing  functional  detail,  the  IDEF0  diagrams  accurately 
represented  the  functional  tasks  and  operations  performed  by  the  pilot. 
Additionally,  the  corresponding  mechanisms  and  controls  which 
interacted  with  tasks  and  operations  were  portrayed  at  various  levels  in  the 
IDEF0  hierarchy.  Taken  together,  these  attributes  comprise  an  analyst’s 
perspective  of  the  mission  and  generate  the  capability  to  perform  analysis 
on  several  aspects  of  the  tactical  air-to-ground  mission. 

3.2.2  Shortcomings.  One  of  the  primary  shortcomings  of  the  IDEFq 
analysis  was  that  decision  making  criteria  are  only  evident,  if  present  at 
all,  at  the  lowest  level  of  the  diagrams.  The  functional  results/outputs  were 
clearly  defined,  but  neither  the  cognitive  processes  nor  the  perceptual- 
action-environmental  interactions  necessary  for  complex  flight  were 
apparent.  The  diagram  provided  a  functional  knowledge  representation 
that  was  oriented  toward  resources  and  results,  but  failed  to  show:  1)  the 
underlying  decision  rules,  criteria,  or  alternatives;  and  2)  the  perceptual 
discriminations  necessary  for  situated  actions. 

Another  significant  weakness  arose  from  the  fact  that  IDEF0is 
structured  with  precisely  defined  boundaries.  This  weakness  becomes  a 
major  problem  for  the  modeling  of  extremely  complex  and  dynamic 


71 


environments  where  boundaries  are  fluid,  and  the  task  priorities, 
conditions,  and  interrelationships  are  continuously  changing. 

Another  difficulty  encountered  with  the  use  of  the  IDEFq technique 
occurred  with  the  representation  of  some  of  the  discrete  parameters 
involved  in  lower  level  functions.  For  a  highly  complex  task  involving  a 
myriad  of  data,  decision  making,  and  possible  outputs,  increasing  levels  of 
detail  were  required  to  create  lower  level  diagrams.  As  the  information 
increased,  the  advantages  of  the  IDEFq  as  a  graphical  information 
representation  was  negated.  When  upper  level  traits  are  replaced  with 
more  detailed  and  explicit  traits,  the  representation  grew  exponentially 
larger,  more  complex,  and  more  cryptic.  As  the  complexity  and  level  of 
abstraction  increased,  the  diagrams  became  more  difficult  to  understand, 
and  the  ability  of  the  expert  (pilot)  to  provide  direct  meaningful  input  was 
impeded.  Consequently,  the  need  to  accurately  describe  a  process  is  in 
direct  conflict  with  the  goal  of  creating  an  easily  understood  graphical 
representation  that  can  be  readily  understood  and  critiqued  by  the  domain 
expert.  The  simplicity  offered  by  pictorial  representation  was  easily 
overwhelmed  by  the  use  of  annotation  and  symbology. 

In  addition,  increased  detail  led  to  unlinked  (tunneled)  parameters  in 
many  cases.  This  tunneling  made  it  necessary  to  audit  <«nd  analyze 
unlinked  parameters  carefully  when  validating  and  verifying  the  IDEFq 
representation.  As  a  consequence,  it  became  difficult  to  independently 
validate  the  information  that  was  represented,  and  for  this  reason,  the 
diagrams  remain  representative  of  the  analyst’s  perspective  of  the  user 
requirements  rather  than  depicting  the  user’s  perspective  of  his  or  her  own 
requirements  for  the  task. 
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4  STORYBOARD  PROTOTYPING:  A  DESIGNER’S  VIEW  OF  THE 
MISSION 


4.0  Design  Acquisition 

As  was  previously  mentioned,  the  design  view  of  the  mission  is  a  very 
critical  component  of  the  overall  framework  for  the  knowledge  and  design 
acquisition  methodology.  Design  acquisition  begins  to  provide  the 
perceptual  basis  for  placing  both  domain  expert  and  knowledge  engineer 
within  the  same  representational  context  that  has  contributed  to  the 
development  of  expertise  by  the  domain  expert.  The  challenge  which  the 
design  acquisition  methodology  has  begun  to  address  involves  the 
innovative  utilization  of  the  pilot  in  the  design  process.  Typically,  pilots 
have  only  been  used  as  reviewers  of  a  previously  established  design 
prototype,  or  knowledge  representation.  During  the  review  process,  the 
pilots  may  be  presented  with  data  or  information  that  is  very  often  conveyed 
to  them  by  means  of  representational  structures  that  are  opaque  and 
difficult  to  comprehend.  On  the  basis  of  these  opaque  representational 
structures,  the  pilots  are  asked  to  comprehend  what  is  being  implemented, 
or  to  predict  the  effects  that  a  new  technology  will  have  on  their  piloting 
behavior,  and  then  provide  their  tacit  approval. 

Kantowitz  &  Sorkin  (1983)  suggest  that  the  first  commandment  of 
human  factors  is  to  “honor  thy  user.”  Simply  having  a  Pilot  Review  Board 
is  not  likely  to  satisfy  the  spirit  behind  the  commandment  of  honoring  thy 
user,  as  it  fails  to  provide  the  opportunity  for  pilots  to  review,  design,  and 
evaluate  in  a  manner  which  is  appropriate  and  natural  for  them.  Hunt 
(1987)  believes  that  to  produce  an  effective  and  efficient  interface,  system 
users  must  be  involved,  in  an  appropriate  manner,  from  the  very  first  step 
of  the  design.  Andriole  (1989)  indicates  that  one  of  the  difficulties  in 
pursuing  this  challenge  is  that  users  are  notoriously  inarticulate  when  it 
comes  to  defining  their  needs.  Design  acquisition  must  begin  to  acquire 
knowledge  from  pilots  in  a  form  that  is  natural  to  their  way  of 
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understanding  flight  requirements  (in  other  words,  knowledge  as  design 
within  a  given  context). 

In  particular,  our  framework  elicits  design  knowledge  by  placing  the 
pilot  in  the  role  of  the  designer  to  design  the  interface  which  connects  the 
pilot  with  his/her  operational  environment.  By  placing  the  pilot  in  this  role, 
the  knowledge  engineer  may  then  learn  the  pilot’s  information 
requirements  by  noting  the  design  attributes  which  pilots  include  in  their 
design  of  an  interface  which  they  consider  suitable  for  flying  a  tactical 
fighter  during  a  given  mission  segment.  Hence,  the  design  view  of  the 
mission  begins  to  unfold  in  iterative  fashion.  Knowledge  transpires  as 
actual  design  attributes  which  arc  spontaneously  evoked  from  a  partial 
representation  of  the  real  world  settings. 


4.1  Storyboard  Tools  and  Techniques 

The  initial  tool  used  to  implement  the  design  acquisition  methodology 
is  “storyboard  prototyping”.  Storyboard  prototyping  can  be  defined  as  an 
interactive  technique  that  weds  multiple  methods  for  requirement 
gathering,  representation,  and  conceptualization  into  a  single  powerful  tool 
for  identifying  and  validating  requirements  before  any  expensive 
programming  needs  to  begin.  The  power  provided  by  storyboard 
prototyping  can  be  described  along  five  facets.  First,  it  provides  the  medium 
to  express  concepts,  ideas,  and  thoughts  as  visual,  auditory,  and  tactile 
designs.  This  replaces  the  semantics  of  the  other  representational  models 
of  the  mission  with  real  world  design  and/or  symbolic  objects.  These  objects 
are  not  language-based  but  are  instead  directly  perceivable  by  the  senses. 
Figure  4-1  illustrates  this  transformation.  Second,  storyboards  provide  the 
type  of  environment  for  erecting,  changing,  evaluating,  and  saving  design 
cases.  The  storyboards  are  housed  on  a  Macintosh/multi-media 
environment  which  provides  playback/slideshow  simulations.  Third,  the 
storyboards  allow  a  rapid  evaluation  of  pilot-vehicle  interface  prototypes. 
This  means  that  pilots  (and  other  design  team  members)  can  easily  and 
inexpensively  iterate  and  modify  the  various  design  prototypes.  A  given 
prototype  can  be  used  as  the  baseline  design  in  order  to  stimulate  the 
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Figure  4-1  Transformation  from  Semantic  to  Isomorphic  Representation 
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exploration  of  divergent  design  solutions.  Fourth,  design  storyboards 
facilitate  problem  identification  for  a  given  design  prototype.  In  this 
situation,  the  user,  problem  type,  and  the  design  become  unconstrained 
variables.  In  other  words,  the  design  storyboard  may  start  with  a  given 
pilot,  a  particular  baseline  design,  and  a  previously  identified  problem,  and 
then  proceed  to  alter  these  factors.  The  intent  is  to  unconstrain  the 
development  of  the  design,  to  include  many  pilots  who  will  generate  many 
designs  and  elaborate  upon  the  problems  that  they  identify  with  other 
design  cases.  The  result  is  what  we  refer  to  as  case-based  design. 

Case-based  design  allows  one  person  to  view  another  person’s  design 
storyboard  to  generate  a  response  as  well  as  his  or  her  own  solution  to  the 
design  problem.  Any  viewer  of  the  design  storyboard  can  document  his  or 
her  rationale  for  any  changes  or  endorsement  of  other  design  cases.  This 
capability  affords  everyone  involved  in  the  design  acquisition  process  the 
opportunity  to  generate  designs  and  provide  explanations  for  their 
particular  model  of  the  world  with  each  explanation  being  directly  linked 
with  the  specific  design  attribute.  Hence,  the  design  and  the  explanation  of 
the  design  can  be  expressed  in  an  explicit  communication  media  in  which 
all  cases  are  available  to  any  individual  utilizing  the  design  storyboard 
utility. 

Because  the  development  of  particular  design  cases  are  accessible  to 
any  pilot  or  other  user  of  the  design  storyboard  utility,  the  characteristics  of 
the  ‘user  model’9  inherent  within  the  PVI  design  becomes  much  more 
explicit.  Vaubel  &  Gettys  (1990)  define  the  user  model  as  an  abstract 
representation  of  the  user  that  models  those  aspects  of  user  behavior  and 
knowledge  needed  by  the  interface.  They  also  suggest  that  the  user  model 
must  be  employed  by  an  adaptive  interface  in  order  to  adapt  itself  to  the 
user’s  needs.  Often  the  model  of  interaction  between  the  pilot  and  the  PA  is 
amorphously  stated  and  unavailable  to  designers,  pilots,  or  analysts.  The 
nature  of  the  representational  models  used  in  this  framework  allow  the 
individual  using  this  utility  to  begin  to  experience  and  understand  the  user 
model  as  it  is  manifest  in  the  various  design  cases.  Because  the  PA  will 

*  User  model  refers  to  human  computer  interface  models  (Card,  Moran,  &  Newell, 
1983),  mental  models  (Johnson-Laird,  1983),  intelligent-student  models  (Sleeman  & 
Brown,  1982),  and  mindware  inferencing  (McNeese,  1986). 
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involve  adaptive  interfaces,  the  experience  of  interacting  with  an  explicit 
user  model  begins  to  expedite  how  the  PA  should  adapt  given:  1)  the  user’s 
concepts,  functions,  needs;  and  2)  the  design  cases  implemented. 

Finally,  the  fifth  facet  of  design  storyboard  power  is  that  the 
representation  simulates  change  in  a  graphic  format  through  a  given  time 
sequence.  One  analogy  might  be  to  look  at  the  storyboard  as  a  comic  strip 
unfolding.  It  has  a  sequence  with  a  specific  content  within  each  frame  that 
tells  a  story.  The  storyboard  shows  something  rather  than  just  describing 
that  something  with  words.  Imagine  reading  the  comics  without  the 
graphic  frames  present.  The  visual  aspects  of  the  comic  strip  are  often 
what  generates  the  humor.  Likewise,  the  storyboard  provides  the  basis  for 
experiencing  the  design  as  opposed  to  simply  reading  a  description  of  the 
design  specification.  The  medium  by  which  the  storyboard  utility  shows 
something  is  not  limited  to  the  visual  modality  (i.e.,  static  graphics,  text 
and  video-based  presentations)  but  may  be  auditory  or  tactile  in  nature.  The 
time  sequence  function  enables  the  storyboards  to  behave  as  a  ‘primitive 
simulation’  providing  the  viewer  with  the  ability  to  see  the  changes  in 
design  elements  that  occur  over  time.  This  adds  a  dynamic  characteristic 
to  the  design  model  and  shows  how  the  interface  must  change  at  different 
phases  of  interaction  with  an  intelligent  system  in  order  to  accommodate 
the  pilot’s  needs. 


4.2  Storvhoarding  the  PA/PVI 

Our  use  of  the  design  storyboard  utility  was  intended  to  supply  a 
designer’s  view  of  the  target  acquisition  phase  of  a  specific  air-to-ground 
mission  (see  Appendix  D)  and  to  evolve  time  sequenced  PVI  design  frames 
(see  Figure  4-2). 

In  order  to  accomplish  this,  pilots  were  placed  in  the  role  of  designers. 
The  intention  was  to  extract  from  the  pilots  what  they  felt  to  be  the 
information  requirements  that  enabled  them  to  transition  between  the 
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Figure  4-2  Storyboard  Time  Sequence 


various  derision  points10  during  the  target  acquisition  phase  of  the  mission. 
In  other  words,  the  pilots  were  asked  to  describe  in  detail  the  information 
they  needed  to  identify  a  particular  derision  point,  as  well  as  the 
information  they  needed  to  control  their  current  course  of  action.  For 
instance,  the  pilots  were  asked  to  describe  the  information  they  used  to 
identify  when  they  had  reached  the  action  point  (one  of  the  decision  points 
between  the  IP  and  WRP)  and  what  information  they  now  needed  in  order 
to  perform  the  actions  that  they  were  initiating  at  this  derision  point. 

Five  of  the  eight  pilots  that  had  been  previously  used  during  the 
concept  mapping  phase  of  the  project  were  used  to  elicit  specific  design 
knowledge.  The  procedure  for  acquiring  design  knowledge  from  pilots 
transpired  after  the  completion  of  the  second  concept  mapping  interview. 
During  the  design  acquisition  phase,  we  continued  to  concept  map  decision 
points  and  information  requirements  that  the  pilots’  identified.  Although 
there  was  some  disagreement  regarding  the  source  of  the  information  that 
was  being  displayed  (owning  to  the  fact  that  the  same  information  is 
frequently  available  from  more  than  one  display  source),  there  was 
complete  agreement  among  the  pilots  regarding  the  specific  decision  points 
between  the  IP  and  WRP  (see  Appendix  A  for  a  concept  map  representation 
of  these  derision  points). 

After  the  pilots  had  identified  the  decision  points  and  elaborated  upon 
the  associated  information  requirements,  the  pilots  were  placed  in  the  role 
of  a  designer.  Until  this  point  in  the  acquisition  process,  a  non-specific 
mission  was  used  in  order  to  establish  the  context.  This  lack  of  specificity, 
it  is  believed,  had  given  the  pilots  the  flexibility  to  discuss  the  concepts  that 
the}'  considered  to  be  most  important  for  the  target  acquisition  phase  of  a 
tactical  air-to-ground  mission.  The  storyboard  utility,  however,  required  an 
increased  level  of  specificity  in  order  to  elicit  design  attributes  which  could 
be  meaningfully  tied  to  a  specific  set  of  environmental  and  mission-related 
conditions.  Thus,  in  order  to  facilitate  the  use  of  the  pilots  in  the  role  of 
designer,  the  pilots  were  given  a  specific  mission  plan  (complete  with  attack 
geometry)  to  review.  This  plan  was  developed  with  the  assistance  of  fighter 

,0  A  decision  point  was  defined  for  the  pilots  as  being  the  point  at  which  they 
transitioned  from  one  course  of  action  to  begin  another  separate  course  of  action.  For 
example,  the  point  at  which  the  pilot  decided  to  stop  a  terrain  following  flight  and  begin  a 
rapid  ascent  would  constitute  a  decision  point. 
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pilots  for  authenticity  purposes.  After  the  pilots  had  reviewed  the 
mission  plan,  they  were  asked  to  design  a  sequence  of  storyboards  for  the 
target  acquisition  phase.  A  separate  storyboard,  containing  the 
information  that  was  required  at  that  point,  was  generated  for  each  of  the 
decision  points  during  the  target  acquisition  phase. 

On  the  basis  of  the  information  that  had  been  gleaned  during  the 
concept  mapping  sessions,  it  was  decided  to  confine  the  storyboard  designs 
to  either  a  full  windscreen  Heads  Up  Display  (HUD)  and/or  a  Helmet 
Mounted  Display  (HMD).  This  was  due  to  the  fact  that  all  of  the  pilots  had 
remarked  that  they  would  like,  if  it  were  possible,  to  keep  their  heads  up 
and  looking  out  of  the  cockpit  during  the  target  acquisition  phase  of  the 
mission. 

The  handcrafted  storyboard  procedure  involved  placing  a  three  foot 
wide  piece  of  paper  in  front  of  each  pilot  for  each  of  the  decision  points.  They 
were  then  asked  to  draw  in  what  they  wanted  to  see  from  the  display 
surface  in  order  to  assist  them  in  the  performance  of  the  mission.  The 
pilots  were  given  the  freedom  to  have  either  a  180°  wide  field  of  view  HUD,  a 
HMD,  or  both.  Pilots  were  continually  asked  to  explain  what  they  were 
doing  and  to  provide  a  rationale  for  each  display  element  created.  The 
pilots  were  also  asked  for  information  that  they  would  prefer  to  have 
presented  in  either  an  auditory/voice,  or  tactile  format.  In  essence,  we 
were  requesting  pilots  to  translate  their  verbal  enunciation  of  information 
requirements  gathered  early  in  the  concept  mapping  session  into  actual 
design  frames  for  each  decision  point  of  the  target  acquisition  phase  of  the 
mission. 

After  each  of  the  pilots  had  participated  in  the  design  acquisition 
session,  their  handcrafted  design  storyboards  were  integrated  in  the  form 
of  an  online  summary  storyboard  using  Silicon  Beach,  Inc.’s  Supercard 
application  software  (Appendix  D  shows  the  summary  storyboard  frames 
which  have  been  developed).  The  summary  storyboard  constituted  a 
compilation  of  the  replicated  ideas  as  well  as  the  inclusion  of  the  innovative 
aspects  taken  from  each  storyboard.  The  summary  storyboard  as  well  as 
individual  pilot  storyboards  were  saved  as  specific  design  cases  for  later 
iterations  of  the  design  acquisition  process.  The  next  step  in  this  process 
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requires  the  summary  storyboard  to  be  validated  by  our  initial  pilots  for 
further  refinements  and  elaborations. 

Once  refined,  the  summary  storyboard  will  be  integrated  with  the 
other  mission  perspectives  derived  from  the  pilot’s  conceptualization  of  the 
mission  (represented  in  the  summary  concept  map)  and  the  use  of  a 
structured  task  decomposition  (e.g.,  the  IDEFo  representation).  Figure  4-3 
illustrates  the  overall  aspects  of  the  design  acquisition  session. 

At  this  point,  the  summary  storyboard  would  be  in  a  state  where  it 
would  be  ready  for  dissemination  to  other  pilots  for  additional  iteration  and 
review,  thereby  enabling  knowledge  as  design  to  evolve. 


4.3  Observations  and  Results 

The  use  of  storyboards  has  reinforced  our  ideas  regarding  the 
importance  of  affording  pilots  the  opportunity  to  translate  their  conceptual 
knowledge  and  expertise  into  a  representation  and  design  prototype  which 
could  be  perceptually  experienced  by  other  utility  users.  Consequently,  one 
observation  which  could  be  made  is  that  the  storyboarding  created  a 
medium  wherein  perceptual  judgments  could  be  shared  or  transmitted 
between  the  knowledge  engineer  and  the  pilot.  For  example,  rather  than 
just  telling  the  knowledge  engineers  about  an  information  requirement  the 
pilot  could  actually  show  him  or  her  by  drawing  a  graphical  representation 
not  only  what,  but  also,  how  they  would  like  to  have  the  information 
portrayed.  The  goal  of  using  knowledge  as  design  was  thus  accomplished 
as  design  acts  to  make  the  knowledge  extant. 

It  was  often  initially  easier  for  a  pilot  (or  for  that  matter,  any  user)  to 
analyze  what  is  wrong/right  with  an  example  design  than  it  was  to 
generate  their  own  designs  from  scratch.  Through  an  exercise  of  analysis, 
synthesis  is  born;  thus,  when  the  pilot  appeared  to  be  having  difficulty 
generating  design  ideas,  he  or  she  was  given  the  opportunity  to  review  and 
critique  a  straw  man  design  (i.e.,  current  aircraft  displays)  as  a  way  to 
begin  the  generation  and  discovery  of  their  own  designs.  This  proved 
successful  as  it  apparently  provided  the  pilots  with  a  direct  opening  into 
their  personal  experience  which  in  turn  gave  them  a  basis  to  drive  their 
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Knowledge  and  Design  Acquisition  Methodology 
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Figure  4-3  Knowledge  and  Design  Acquisition  Methodology 


design.  The  pilots  tended  to  generalize  from  their  experience  in  order  to 
create  designs  which  would  more  effectively  satisfy  their  own  information 
requirements.  In  all  instances,  the  pilots  were  able  to  transition  from 
analysis  to  synthesis  in  order  to  complete  the  design  task.  These 
observations  have  led  us  to  the  conclusion  that,  when  given  an  opportunity 
and  the  proper  set  of  tools,  pilots  were  able  to  perform  effectively  in  the  role 
of  designers. 

Often  the  argument  is  made  that  pilots  only  have  a  limited 
understanding  of  technological  advances  and  consequently  would  fail  to 
know  what  is  required  for  the  PA.  Our  results  suggest  that  pilots  are  richly 
endowed  with  what  they  need  for  particular  aspects  of  the  mission  in  terms 
which  would  help  them.  They  were  able  to  communicate  these  needs,  and 
in  many  cases,  design  the  interface  themselves.  We  believe  this  is  the 
correct  approach  to  the  design  process.  Technology  should  not  drive  the 
pilot;  but  the  pilot,  on  the  basis  of  his  or  her  needs,  should  be  the  main  force 
which  drives  the  development  of  the  PA.  The  point  is  to  preserve  and 
represent  these  needs,  such  that  they  can  be  given  to  other  design  members 
for  input.  Concomitantly,  the  pilot,  interacting  within  this  knowledge  and 
design  acquisition  framework,  should  have  access  to  other  PA  team 
member  views  of  the  mission.  In  this  way,  learning  becomes  paramount 
and  an  active  part  of  design;  design  ideas  are  distributed  across  the  PA 
team  and  act  as  generators  of  potential  knowledge/design  acquisition  to 
create  large-scale  knowledge  bases  suitable  for  deriving  the  PA. 

In  this  design  acquisition  session,  it  was  also  observed  that  perceptual- 
based  design  is  a  main  driver  for  an  integrative  design  structure  tying  all 
the  mission  views  and  representations  of  those  views  together.  We  observed 
that  as  a  part  of  explaining  or  providing  rationale  for  a  design  element,  the 
pilot  naturally  established  links  to  their  concept  maps,  both  in  terms  of 
information  requirements  and  as  a  guidepost  which  fueled  the  design 
generation  process.  This  natural  connection  between  the  semantic  and 
perceptual  representations  of  knowledge,  gauged  in  accordance  with 
specific  decision  points  and  information  requirements,  underlies  the 
integration  of  the  disparate  knowledge  representations.  When  pilots  begin 
to  make  the  connections  between  their  thoughts  and  the  way  these  thoughts 
should  be  transformed  into  designs  as  they  perform,  a  sound  basis  for 
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configuring  a  PA  has  been  established. 

An  intelligent  system  must  be  able  to  interface  with  the  pilot  in  terms 
of  both  his/her  general  and  specific  knowledge  about  the  given  phase  of  the 
mission,  as  well  as  the  cockpit  design  itself.  The  software  which  underlies 
the  PA  must  capture  multiple  views  of  how  a  pilot  envisions  the  mission  as 
well  as  the  associate  itself.  The  power  of  the  knowledge  and  design 
acquisition  methodology  is  in  providing  an  interactive  medium  that  directly 
connects  the  pilot  to  the  design  of  an  intelligent  sytem  which  will  provide 
him/her  decision  support.  By  capturing  the  pilot’s  knowledge  and  design, 
the  PA  has  obtained  a  ‘mental  model’  and  ‘pilot  interface’  baseline  for 
further  development. 


5  TOWARD  AN  INTEGRATIVE  STRUCTURE 


5.0  The  Integration  of  Three  Different  Knowledge  Representations 

The  advanced  knowledge  and  design  acquisition  framework  has  led  to 
three  distinct  representations  of  the  target  acquisition  phase  of  the  air-to- 
ground  mission:  concept  maps,  IDEF  diagrams,  and  design  storyboards,  as 
propagated  from  direct  pilot  input.  One  of  the  goals  of  this  effort,  as  it 
continues,  is  the  development  of  an  integrative  knowledge  structure  within 
the  domain  of  a  hypermedia  environment.  This  structure  would  allow  the 
establishment  of  the  inter-connective  linkages  across  the  three  knowledge 
representations,  and  provide  the  means  for  understanding  the  impact  that 
the  elements  of  one  representation  would  have  upon  the  elements  of  the 
remaining  two  representations.  The  proposed  structure  involves  the 
creation  of  linear  and  non-linear  associations  among  the  various  elements 
of  knowledge  representation.  The  intent  is  to  allow  any  given  user  of  this 
utility  (e.g.,  a  human  factors  engineer,  a  system  designer,  a  pilot,  etc.)  to 
gain  direct  access  to  multiple  knowledge  representations. 

This  access  to  the  knowledge  representations  would  enable  the  user  of 
the  utility  to  both  navigate  between  the  different  forms  of  knowledge  by 
means  of  the  existing  linkages  and  establish  new  linkages  when  deemed 
appropriate.  Whenever  a  link  is  traversed  or  created  the  user  would  also 
have  the  option  of  adding  text  comments  for  rationale,  questions,  or 
explanation.  After  a  user  has  completed  a  session  in  the  integrative 
structure  their  changes,  additions,  and  search  through  the  structure 
would  be  saved  as  a  new  case  of  the  integrative  structure.  As  the  number  of 
users  increases,  the  number  of  linkages  between  the  representations  will 
increase,  thereby  fostering  a  corresponding  increase  in  the  breadth  and 
depth  of  the  actual  knowledge  base. 


5.1  Navigating  Linkages 


Given  that  there  are  three  representational  schemes  within  the 
integrative  structure,  the  user  would  be  able  to  enter  the  structure  at  any 
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given  i;'i'iRiTt  wiihin  a  scheme.  If  an  element  in  a  knowledge 
.cotv'sentation  (i.e.,  such  as  a  concept  from  within  a  concept  map)  has 
already  been  connected  to  another  element  of  knowledge  (i.e.,  a  design 
element  from  within  a  design  storyboard),  that  linkage  would  be  identified 
as  existing  and  available  to  the  user  (e.g.,  with  a  button  or  hot  spot).  When 
the  button  is  activated,  the  user  would  be  transported  from  the  first 
knowledge  representation  to  the  second  associated  knowledge 
representation  as  a  way  of  demonstrating  the  relationship  that  exists 
between  the  two  representations. 

For  example,  if  the  user  of  the  utility  were  examining  the  design 
storyboard  produced  by  one  of  the  pilots,  the  user  would  be  able  to  move  from 
a  particular  design  element  in  the  storyboard,  to  the  related  concept  within 
a  concept  map.  This  journey  from  the  storyboard  element  to  the  concept 
map  would  presumably  provide  the  user  with  the  insights  (or 
conceptualizations)  that  led  to  the  inclusion  of  that  design  element  within 
the  storyboard.  For  instance,  if  the  user  were  interested  in  finding  out  why 
a  visual  offset  point  had  been  included  in  the  storyboard,  he  or  she  could  be 
directly  transported  to  the  portion  of  the  pilot’s  concept  map  which 
discussed  the  limitations  of  the  target  designator  box,  and  the  need  to  have 
an  additional  visual  referent  (the  offset  point)  that  would  be  tied  to  an  easily 
identifiable  environmental  feature. 


5.2  Establishing  Linkages 

Using  this  integrated  structure  would  enable  the  viewer  to 
simultaneously  view  multiple  knowledge  representations  and  establish 
links  among  elements  of  the  disparate  representations.  Note  that  in  the 
process  of  establishing  linkage,  the  user  has  the  option  to  document  any 
rationale,  guidance,  or  comments  associated  with  the  link.  These  “link 
characterizations”  would  be  saved  as  part  of  that  user’s  case,  and  would 
then  become  accessible  to  other  users  of  the  integrative  knowledge 
structure. 

Figure  5-1  provides  a  graphic  illustration  of  the  integrative  structure 
in  terms  of  the  overall  knowledge  space  available  for  conceptualization, 
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IDEF  Activities 


Figure  5-1  Integrative  Structure 
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analysis,  synthesis,  research,  and  design  opportunities.  The  cube  shows 
the  union  of  the  three  different  types  of  knowledge  representations  acquired 
(e.g.,  storyboard  design  cases,  concepts,  and  IDEF0  activities).  The  check 
mark  indicates  the  existence  of  a  specific  link  between  any  two  elements 
within  a  given  knowledge  representation  (e.g.,  a  check  mark  links  element 
A4  and  C4  as  a  partially  integrated  substructure).  The  cube  that  is 
illustrated  in  Figure  5-1  shows  only  four  elements  per  knowledge 
representation  for  sake  of  example,  but  there  may  be  as  many  elements  as 
required  in  each  representational  scheme.  For  example,  a  concept  map 
may  have  as  many  as  500  concepts  or  more,  with  each  concept  or  cluster  of 
concepts  depicted  as  an  element  within  the  integrative  structure. 

One  way  to  explain  the  integrative  structure  is  to  describe  its  various 
building  blocks  and  the  way  that  each  block  exists  in  a  hierarchical  relation 
with  another  block.  Figure  5-2  lists  these  building  blocks. 

At  the  most  global  level,  the  overall  knowledge  space  represents  the 
union  of  the  three  knowledge  representations.  Second,  each  representation 
contains  a  number  of  specified  elements  (i.e.,  the  concepts  within  a  concept 
map,  the  activity  boxes  within  an  IDEF0  diagram,  or  the  design  objects  of  a 
design  storyboard).  At  a  more  local  level,  each  element  may  consist  of 
specific  attributes.  The  attributes  can  be  most  easily  understood  with 
reference  to  the  IDEF0  model,  in  which  the  inputs,  mechanisms,  controls, 
and  outputs  associated  with  an  activity  box  are  the  attributes  of  that 
element. 

Because  of  the  lack  of  predefined  structure  associated  with  the  concept 
map,  its  attributes  must  be  defined  relatively.  In  other  words,  whether  a 
concept  is  an  element  or  an  attribute  of  that  element  is  likely  to  change  as  a 
function  of  the  viewer’s  focus.  The  attributes  of  a  concept  are  simply  other 
concepts  that  qualify  or  serve  to  define  the  properties  of  the  former  concept. 
For  example,  if  the  key  concept  of  a  concept  cluster  has  been  defined  as  an 
element  of  the  concept  map,  then  the  various  concepts  in  the  cluster  could 
be  regarded  as  the  attributes  of  that  key  concept.  However,  any  one  of  the 
various  concepts  in  the  cluster  may  also  function  as  parent  concept  further 
defined  by  additional  concepts.  In  this  case,  the  parent  concept  would  be 
regarded  as  the  element  with  the  child  concepts  being  its  attributes. 
Whenever  the  concept  map  undergoes  a  “rubber  sheet”  transformation,  the 
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(  BUILDING  BLOCKS  ) 


•  Knowledge  Space  =  U  {  representations  A,  B,  C  } 

•  Representation  A,  B,  C  =  U  {  elements  1  to  x } 

•  Elements  x  =  U  {attributes  within  representations  A,  B,  C  } 


-examples  IDEF:  inputs,  mechanisms,  controls,  outputs 

CM:  clusters,  relations,  individual  pilot  maps,  individual  pilot 
interview  tapes 

Design  Cases:  frame,  decision  point-lime  sequence,  design  symbology,  HF  features 


-special  attributes:  the  IDEFo dictionary 
the  KEY  CONCEPTS 


Figure  5-2  Building  Blocks  for  an  Integrative  Structure 
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relative  aspects  of  the  attributes  also  change.  Concepts  which  were  once 
children  may  then  be  repositioned  in  the  map  as  parents.  Hence,  the 
attribute  structure  changes  accordingly.  Consequently,  the  particular 
rubber  sheet  ‘states’  may  act  as  attributes  of  relativity. 

The  attributes  of  a  given  design  case  also  exist  but,  like  the  concept 
map  attributes,  the  storyboard  attributes  are  not  as  grammatically  explicit 
as  IDEF0  attributes.  The  design  objects  within  a  storyboard  frame  are  likely 
to  be  the  elements  with  its  attributes  being  such  things  as:  the  design 
symbology,  the  spatial-temporal  locations,  the  required  switchology,  and  the 
specific  human  factors  engineering  codification  features  (e.g.  color,  display 
luminance,  and  auditory  signatures).  Hence,  all  three  types  of  knowledge 
representations  are  comprised  of  elements  with  various  attributes 
composed  in  different  formations. 


5.3  Linkage  Types 

The  integrative  structure  proposes  three  major  types  of  linkages: 
simple,  compound,  and  complex  connections  of  entities  (see  Figure  5-3). 

Simple  linkages  represent  the  intersections  between  two  or  more 
specifically  defined  elements  (or  attributes)  of  different  knowledge 
representations.  Simple  linkages  are  further  defined  as  having  two 
different  types  of  links,  primary  links  and  secondary  links  which  are 
differentiated  on  the  basis  of  whether  they  connect  two  or  more  elements  (or 
attributes)  that  exist  on  the  same  level  of  granularity,  or  whether  they 
connect  elements  (or  attributes)  that  traverse  different  levels  of  granularity. 
The  primary  linkages  represent  the  intersection  of  specifically  defined 
elements  (or  attributes)  from  two  or  more  knowledge  representations  that 
exist  at  comparable  levels  of  granularity.  For  instance,  a  given  concept  of  a 
map  (e.g.,  “bomb  fragmentation  altitude”)  may  be  linked  to  a  design 
storyboard  element  (e.g.,  the  graphic  depiction  of  the  weapon  delivery 
clock).  The  secondary  links  are  those  that  connect  elements  or  attributes 
that  arc  at  distinctly  different  levels  of  granularity  within  their  respective 
knowledge  representations.  For  example,  the  concept  mapping  knowledge 
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SIMPLE  LINKAGES 

> 

primary  links 

*  ni 

{ x  units  of  any  2  or  more  dimensions  } 

secondary  links 

*  n2 

{ x  trace  dimensions  of  any  two  or  more 
dimensions 

OR 

*  ^3 

{x  trace  dimensions  of  any  x  unit(s)  of  a 
dimension 

COMPOUND  LINKAGES 

•  n4 

{  2  or  more  partial  structures} 

COMPLEX  LINKAGES 

•  n5 

{2  or  more  fully  integrated  structures} 

•  n6 

{x  partial  structure(s)  and  x  ull  structure(s) 

Figure  5-3  Linkage  Types 
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representation  element  “situation  awareness”  might  be  linked  to  the  IDEFq 
attribute,  “navigation  system  malfunction  status”. 

When  simple  linkages  connect  just  two  elements  and/or  attributes 
together,  a  partially  integrated  substructure  is  formed  (see  Figure  5-4). 
When  simple  linkages  connect  all  three  knowledge  representations,  then  a 
fully  integrated  substructure  is  created  (see  Figure  5-5).  The  fully 
integrated  substructure  reflects  a  volume  of  the  overall  knowledge  space 
and  consists  of  the  integration  of  three  partial  substructures  (which  reflect 
only  a  surface  on  the  knowledge  space).  The  formation  of  fully  integrated 
substructures  begins  to  signify  the  progressive  deepening  of  knowledge. 

The  next  major  type  of  connection,  compound  linkages,  consists  of  the 
intersection  of  two  or  more  partial  substructures  that  traverse  levels  of  the 
hierarchical  structure.  This  may  signify  the  composition  of  knowledge 
from  more  basic  levels  to  more  generalized  relationships.  The  compound 
linkage  also  takes  an  expansive  view  of  complexity  in  that,  by  definition,  the 
network  of  relationships  is  expanding.  This  may  also  be  the  basis  for 
analogical  renderings  of  acquired  knowledge. 

The  last  connection  type,  complex  linkages,  consists  of  the  intersection 
of  two  or  more  integrated  substructures,  or  the  intersection  of  a  partial 
substructure(s)  with  a  full  substructure(s).  Like  the  compound  links, 
complex  links  represent  the  most  generalized  and  expansive  forms  of 
integration  within  the  knowledge  space.  As  several  of  these  links  get 
connected  together,  the  basis  for  rule  derivation,  causality,  and  belief 
systems  may  be  formed.  Hence,  it  is  evident  that  there  are  a  number  of 
different  types  of  linkages  possible  within  the  integrative  structure. 


5.4  Evolution  of  the  Integrative  Knowledge  Structure 

The  discussion  of  integrative  structures  has  assumed  that  relations 
within  a  given  representation  have  already  been  specified  for  that  given 
representation.  Within  the  IDEF0  diagram,  for  example,  there  are  already 
specified  relations  between  attributes  and  the  elements  (e.g.,  how  controls 
effect  flow  from  input  to  output  given  a  certain  defined  activity).  However, 
as  new  users  examine  the  integrative  structure  they  may  see  additional 
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within-representation  relations  which  hitherto  were  not  identified.  The 
integrated  structural  utility  will  allow  such  within-dimension  relations  to 
be  made  as  well,  thereby  facilitating  the  maturation  of  the  individual 
representations  of  knowledge.  It  is  also  possible  that  by  adding  new 
associations,  generative  activity  will  occur  which  spawns  entire  new 
directions  of  knowledge/design.  So  if  a  user  makes  a  linkage  and  it  creates 
a  major  insight,  the  possibility  for  expansion  of  any  view  is  facilitated. 
Hence  the  integrative  structure  exists  as  a  possibility  for  further 
development. 

This  section  has  described  the  knowledge  space,  building  blocks,  and 
specific  linkages  which  form  the  cornerstones  of  the  integrative  structure. 
What  has  not  been  elaborated  is  who  makes  the  links?  The  design  team 
members  are  the  people  who  are  entrusted  with  the  job  of  establishing  new 
linkages  as  a  part  of  the  knowledge  and  design  acquisition  process. 
Whenever  a  linkage  is  made  within  the  domain  of  the  integrative  structure, 
it  would  be  tagged  as  to  the  perspective  of  the  individual  design  team 
member  effecting  the  connection.  Therefore,  in  addition  to  each  knowledge 
representation  being  indicative  of  a  different  perspective,  any  changes  to 
these  models  or  links  among  these  models  will  also  be  traceable  to  a 
particular  individual’s  perspective  (i.e.,  designer,  pilot,  human  factors 
engineer,  etc.). 

The  intention  for  ‘link  production’  is  to  have  the  same  pilots  whose 
knowledge  is  embodied  in  the  current  representations  establish  the  initial 
linkages  between  those  various  representations.  In  this  way,  the  pilots 
would  be  making  links  among  entities  which  in  fact  they  generated  in  their 
earlier  sections.  An  observation  during  the  storyboard  session  revealed 
that  the  pilots  were  quite  comfortable  moving  back  and  forth  between  the 
concept  maps  and  the  design  storyboards  and  establishing  the  connections 
between  these  two  knowledge  representations.  We  found  that  as  pilots 
began  to  design  their  storyboards  they  also  would  talk  about  concepts  and 
functions  required. 

The  set  of  baseline  simple  linkages  would  represent  the  first  order  of 
growth  within  the  integrative  structure.  As  the  integrative  structure  is 
viewed  from  the  different  perspectives,  the  level  of  learning  and  the  amount 
of  knowledge  progressively  deepens  to  yield  more  complex  relationships. 
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The  formation  of  multiple  integrated  substructures  within  the  knowledge 
space  signals  that  substantial  learning  and  integration  is  taking  place  as 
the  different  models  of  knowledge  (which  were  initially  disparate  and 
isolated)  become  inter-related  and  usable  for  future  knowledge  and  design 
acquisition  activities.  In  essence,  the  integrative  structure  takes  the 
‘mental  models’  of  many  and  explicitly  provides  them  to  others  to  look  at, 
learn  from,  adapt,  and  decimate  if  need  be.  The  vision  of  what  the  P4  must 
consist  of,  what  it  produces,  and  how  it  works  are  a  function  of  knowing 
how  pilots  think  about,  design,  and  use  current  cockpit  technology, 
obtaining  their  ideas  on  where  weaknesses  are,  and  seeking  their  vision  as 
to  what  informalion/functions  they  require.  As  a  consequence,  the 
development  of  the  PA  is  in  direct  association  with  the  pilot  such  that  the 
pilot  drives  the  design  rather  than  having  the  pilot  serve  the  technology. 
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6.0  SUMMARY  \NP  CONCLUSION 


The  identification  of  aircraft  system  requirements  has  traditionally 
been  accomplished  during  the  initial  phases  of  the  engineering  design 
process.  To  a  large  degree  these  requirements  are  influenced  by  the 
designer's  experience  level  and  ability,  and  are  driven  by  technical 
constraints  that  result  from  the  various  hardware  and  software  limitations 
and  capabilities.  This  process  all  too  frequently  results  in  a  mismatch 
between  what  the  pilots  need,  and  what  the  engineers  designed  (Norman, 
1988;  Zaff,  1989).  The  elimination  of  resulting  discrepancies  typically 
requires  a  time  consuming  and  expensive  process  of  iterative  redesign  and 
refinement  based  on  pilot  testing  and  critique.  In  order  to  facilitate  the 
design  of  a  system  that  more  closely  matches  the  needs,  capabilities  and 
desires  of  the  user,  we  have  advocated  an  approach  to  system  design  and 
development  that  attempts  to  fuse  together  the  disparate  perspectives  of 
user  requirements  arising  from  the  system  analyst’s  conception  of  the 
requirements,  the  designer’s  conception  of  the  requirements,  and  user’s 
conception  of  the  requirements.  In  order  to  accomplish  this  goal,  we  have 
set  out  to  investigate  the  requirements  definition  process  from  all  three 
perspectives,  and  to  capture  the  requirements  with  an  integrated  set  of 
knowledge  and  design  acquisition  techniques  that  facilitate  graphical 
knowledge  representation  and  iterative  design. 

The  efforts  to  develop  a  knowledge  and  design  acquisition  methodology 
has  placed  great  emphasis  on  obtaining  information  from  pilots  to 
demonstrate  the  utility  for  capturing  the  pilots’  perspective  on  the  mission 
requirements.  The  methodology  appears  to  be  capable  of  providing  the 
eventual  large-scale  knowledge  base  which  has  been  envisioned  for  the 
Pilot’s  Associate.  For  this  reason,  the  integrated  knowledge  and  design 
acquisition  process  gave  the  pilots  the  opportunity  to  be  placed  in  the  roles  of 
men'  or,  analyst,  and  designer.  It  also  placed  knowledge  engineers  in  the 
role  of ‘apprentices’  to  the  pilots  This  was  a  unique  utilization  of  the  pilots’ 
knowledge  in  the  design  process,  as  normally  they  are  only  asked  for  their 
opinions  during  a  cursory  review  of  the  design  prototypes. 

Finally,  the  vision  of  transforming  knowledge  as  design  has  been 
demonstrated  as  an  innovative,  productive,  and  viable  goal  for  acquiring, 
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assimilating,  and  representing  knowledge  within  the  application  of  the 
Pilot’s  Associate.  The  manifestation  of  knowledge  as  design  facilitates  the 
realization  that  pilots  can  in  fact  be  integral  in  the  development  of  a 
cooperative  knowledge  based  system  proposed  to  be  their  ‘associate’. 
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APPENDIX  A 
CONCEPT  MAPS 
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Concept  Map  Legend 


Segmented  Concept  Map  l1 


Complete  Concept  Map  1 
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Complete  Concept  Map  2 
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Segmented  Concept  Map  2 
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pg  11C 
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pg  H9 

In  practice  the  concept  maps  arc  viewed  on  a  single  continuous  sheet  of  paper.  Printing  restrictions 
have  prevented  its  presentation  in  this  more  dcsircrable  format. 
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USED  AT:  AUTHOR:  J  DUNCAN  r  ,-fE:  7/10/90  x  WORKJNG  READER  DATE  CONTEXT: 

PROJFCT  PILOT  ASSOCIATE:  AIR  GROUND  REV:  DRAFT 
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C  onlcxl:  Perform  lacucal  Air  -Ground  Mission,  Emphasis  on  Ingress-  large!  AtlatK- Egress  Mission  si 
Viewpoint:  Pilot  Task  Requirements 

Purpose:  Develop  Descriptive  Model  of  Tactical  Air-Ground  Mission  to  be  used  for 

1)  KA  guidance  in  Pilot  Information/Dccision  Analysis 

2)  Basis  for  Computer  Simulation  of  Mission  Execution  and  Pilot  Information  Requirements 
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NODE:  A3  I  TITLE:  OFFENSIVE  ACTIONS  NUMBER: 
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NODE:  A3 1  TITLE:  PERFORM  OFFENSIVE  AIR-AIR  NUMBER: 
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Aircraft  Systems 
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USED  AT:  AUTHOR:  J.  DUNCAN  DATE:  7/10/90  x  WORKING _  READER _ DATE  CONTEXT: 

PROJECT:  P'LOT  ASSOCIATE:  AIR  GROUND  REV:  DRAFT  Dj-j 

|  RECOMMENDED _ Dn 

NOTES:  123456789  10  PUBLICATION  _ _ 1 

Cl  Mission  Plan  C2  Technical  Orders  CJ  Doclrinc/KUh  C4  A/C  Safety 
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Ml  Pilot  M2  Aircraft  Systems 

NODE.  A32I2I  TITLE:  CONFIGURE  FIRE  CONTROL  &  WEAPON  SYSYTEMS  j  NUMBER: 
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NODE:  A32124  TITLE:  CONFIGURE  NAVIGATION  SYSTEM  NUMBER: 
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NODF:  A32125  TITLE:  CONFIGURE  COUNTERMEASURE  SYSTEMS  NUMBER: 
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NUMBER: 


NODE:  A3221  TITLE:  ACQUIRE  TARGET 
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USED  AT:  AUTHOR:  J.  DUNCAN  DATE:  7/10/90  x  WORKING _  READER _ DATE  CONTEXT: 

PROJECT:  P|LOT  ASSOCIATE:  AIR-GROUND  REV:  DRAFT _ □ 

RECOMMENDED _ *  □ 

NOTES:  123456789  10  PUBLICATION 

Cl  Mission  Plan  C2  Technical  Orders  C3  Doctrine/ROE  C4  A/C  Safety  C5  Target  Damage 


NODE:  A3222  TITLE:  RELEASE  WEAPONS  NUMBER: 
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M2  Aircraft  Systems 
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NODE:  A42  TIME:  ASSESS  MISSION  PHASES  NUMBER: 


APPENDIX  C 

IDEFq  data  dictionary 
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AEP-ENPiX-C 


IDEF  DATA  DICTIONARY  -  TACTICAL  AIR-GROUND  MISSION 

The  IDEF  Tactical  Air-Ground  Mission  comprises  a  pilot  oriented 
model  of  an  air-ground  mission  for  usn  in  analysis  of  the  pilot  tasks, 
performance  and  information  requirements  necessary  to  accomplish 
the  mission.  An  emphasis  is  placed  on  the  associated  mission 
functions  occurring  during  the  ingress  to  the  target  IP  phase,  the 
target  attack  phase,  and  the  egress  phase.  The  model  is  based  on 
mission  performance  by  an  F-16  type  aircraft  with  CCIP  and  CCRP 
weapon  delivery  modes  and  comparable  aircraft  fire  control,  radar, 
and  related  weapon  systems  accuracy  and  capabilities. 

The  IDEF  Data  Dictionary  provides  definitions  for  each  function 
within  the  IDEF  Tactical  Air-Ground  Mission  decomposition  and 
unique  Inputs.  Outputs,  Controls/Constraints,  and  performance 
Mechanisms  are  described.  A  description  of  pertinent  parameters  is 
provided  for  those  functions  requiring  elaboration. 


A-0  PERFORM  TACTICAL  AIR-GROUND  MISSION 

This  function  represents  the  performance  of  a  Tactical  Air-to- 
Ground  Mission.  The  various  functional  parameters  represent  the  top 
level  requirements  for  aircraft  operation  and  mission  performance 
for  all  aspects  of  the  mission. 
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INPUTS 


11  Communications,  Sensor  &  Sensory  Inputs 

Inputs  obtained  by  the  pilot  via  one  or  more  of  the  five  human 
senses  :  sight,  touch,  taste,  smell  and  hearing.  For  the  Tactical  Air- 
Ground  Mission  these  include  visual,  aural,  touch,  acceleration 
forces,  vibration,  and  physiological  inputs  to  the  pilot. 

12  Mission  Plan 

The  overall  design,  method,  or  scheme  for  accomplishing  the 
Tactical  Air-Ground  Mission  to  attain  the  mission  objective. 


MECHANISMS 

Ml  Pilot 

Experienced,  combat  ready,  aircrew  position  rated,  USAF  flight 
personnel  responsible  for  performing  the  tactical  air-ground 
mission.  Mission  pilot  task  performance  based  on  current  ability  and 
capabilities  due  to  physiological  condition  as  affected  by  G  loads, 
illness,  fatigue,  and  incapacity  due  to  chemical  exposure  effects  and 
wounds. 

M2  Aircraft  Systems 

The  combination  of  components  -  such  as  the  flight,  propulsion, 
navigation,  defensive,  and  other  applicable  aircraft  systems  -  which 
function  as  an  integrated  system  to  accomplish  the  mission  at  an 
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acceptable  level  of  risk  and  pilot  workload.  Includes  aircraft 
capabilities  as  affected  by  systems  configuration,  fuel  and  stores 
state,  malfunctions,  battle  damage,  system  performance  levels  and 
aircraft  energy  state. 

QU-TEUIS 

01  Mission  Results 

The  cumulative  status  (i.e.  aircraft  position,  systems 
operations/malfunctions,  target  acquisition  and  engagement,  weapon 
effects)  of  each  of  the  mission  segments  throughout  the  mission. 
Individual  actions,  conditions,  and  events  that  culminate  in  the  total 
mission  effectiveness  measures  of  target  disposition  and  aircraft 
survivability. 


Cl  Mission  Plan 

The  overall  design,  method,  or  scheme  for  accomplishing  the 
Tactical  Air-to-Ground  Mission  and  to  attain  mission  objectives. 
Includes  the  flight  route,  planned  procedures,  alternate  actions, 
mission  goals  and  other  information  and  parameters  pertinent  to  the 
mission . 

Mission  Considerations 

Minimize  Exposure  Time 

Limit  Real  Time  Tasks  During  Attack  Phase 
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Maximize  Terrain  Obscuration 
Limit  Pop-up/Egress  Altitude 
Minimize  Altitude 
Maximize  Airspeed 

Maintain  Situation  Awareness 
Aircraft  Status 
Aircraft  Position 
Terrain,  Obstacle  Clearance 
Target  Location 
Threat  Location  &  Status 
Attack  Package  Locations 

Defensive  Response  to  Chemicals 

Attack  Package 
Size,  Mix 

Target  Type 

Level  of  Damage  Required 
Anticipated  Threat 
Asset  Availability 
Expense 

Required  Support 
Communications 

Formation,  Spacing  Requirements 
Delivery  Tactics 

Release  Points 
Frag  Damage 
Spacing 

Attack  Sequence,  Coordination 
Mutual  Support 

Target  Parameters/Characteristics 
Target  Features 

Location  -  Latitude,  Longitude,  Elevation 
Orientation,  Spacing 
Surrounding  Terrain 
Anticipated  Defenses 

Minimize  Exposure  to  Threat 
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Counter/Suppress  Defenses 
Required  Damage  Levels 
Attack  Axis 
Re-Attack  Plans 
Alternate  Target  Plans 
Targets  of  Opportunity 
Target  Vulnerability/Hardness 
Target  Area  Weather 

Cell/Command  Operations 
Coordination 
Communications 
Weapon  Frag  Clearance 
Formation  Spacing 
Defensive  Requirements 
Offensive  Abilities/Mutual  Support 
Situation  Awareness 
Timing 
Weather 

Weapon  Selection 

Type  of  Target 

Level  of  Destruction  Required 
Platform/Weapon  Accuracy 

Aircraft  Fire  Control  System  Capabilities 
Release  Parameters 
Drag  Options 
Type  of  Munition,  Yield 
Lethal  Radius 

Fuzzing/Arming  Requirements 
Timing 

Proximity  -  Burst  Height 
Impact  Angle 
Release  Altitude 
Defenses 
Weather 

Cel!  Attack  Tactics 
Aircraft  Configuration 
Weight 
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Drag 

Aircraft  Performance  &  Range  Degradation 

Availability 

Expense 

Carriage  Capability 

Weapon  Employment  Envelope 
Delivery  Mode 
Minimum  Attack  Perimeter 
Weather 

Release  Parameters 

Interval/Footprint 
Release  Timers 
Dive  Angle 
Airspeed 

Weapon  Impact  Angle 
Release  Altitude 

Terrain,  Obstruction  Clearance 

Altitude  Loss  During  Pull-out 
Minimum  Ground  Clearance 
Altimeter  Lag 
Target  Elevation 
Threat  Exposure 
Own  ship  Frag  Damage 
Cell  Frag  Damage 
Secondary  Explosions 
Seeker  Limits 
Type  of  Weapon 
Drag  Options 
Fuzzing  Requirements 
Time 

Altitude  (Proximity) 

Type  of  release 
Single 
Triple 

Multiple/Ripple 

Threat  Reaction 

Air-Air  Threats 
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Air-Ground  Threats 
Time  Restraints 
Refueling,  Support  Coordination 
Alternate  Targets 
Abort  Criteria,  Procedures 
Emergency  Procedures 
Downed  Aircraft  Procedures 
Recovery  Procedures 


C2  Technical  Orders 

Air  Force  publications  that  give  specific  technical  directions 
and  information  with  respect  to  the  inspection,  operation,  and 
maintenance  of  aircraft  equipment  and  weapons  systems.  Includes 
performance  capabilities  and  limitations  of  all  aircraft  systems. 

C3  Doctrine/Rules  of  Engagement  (ROE) 

The  rules,  propositions,  operational  methods,  tactics  and 
teaching  doctrines  that  have  official  sanction  or  authority  to  be 
used  to  guide  and  direct  pilot  actions  during  the  Tactical  Air-Ground 
Mission. 

Engagement  Tactics 

Weapons  Selection,  Employment 

Operational  Procedures  and  Restrictions 


C4  Aircraft  Safety 

Those  items  directly  related  to  aircraft  survivability  and 
requiring  immediate  attention  to  avoid  pilot  injury  or  fatality. 
Conditions  that  require  corrective  actions  that  take  immediate 
precedence  over  current  actions  and  require  the  interruption  and 
preemption  of  current  tasks,  due  to  the  impending  destruction  of  the 
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aircraft  due  to  threat  activity,  catastrophic  systems  failures, 
ground/obstruction  clearance,  collision  avoidance,  or  pilot 
physiological  condition/incapacity. 

Items  of  concern  include: 

Threat  Status 

Terrain  Clearance 

Obstacle  Avoidance 

Pilot  Reaction  to  Chemical  Exposure 

Pilot  Wounds 

Pilot  G  Tolerance/Capability  -  G  Induced  Loss  of 
Consciousness  (GLOC) 

Pilot  Illness,  Fatigue 

Aircraft  Status-  Structural  Integrity, Energy  State, 
Fuel  State,  Systems  Status 
Collision  Avoidance 


AO  PERFORM  TACTICAL  AIR-GROUND  MISSION 

A1  CONTROL  AIRCRAFT 

Pilot  performance  of  those  actions  that  involve  controlling 
aircraft  flight  parameters,  navigation,  communications,  and  aircraft 
systems  monitoring  and  management. 

A2  DEFEND  AIRCRAFT 

Pilot  performance  of  those  actions  that  involve  the  defense  of 
the  aircraft  against  ground  based  and  airborne  threats. 

A3  OFFENSIVE  ACTIONS 

Pilot  performance  of  those  offensive  actions  that  involve  the 
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use  of  aircraft  munitions  against  ground  based  and  airborne  targets. 


A4  MANAGE  MISSION  PHASES 

Pilot  evaluation  of  current  mission  status  based  on  aircraft 
system  performance,  target  status,  fuel  and  stores,  aircraft 
position,  environmental  status,  mission/phase,  and  threats  status  to 
determine  the  current/subsequent  course  of  action  and  transition 
between  various  mission  phases. 


A1  CONTROL  AIRCRAFT 

All  FLY  AIRCRAFT 

Perform  those  actions  that  involve  controlling  aircraft  flight 
parameters  and  executing  flight  procedures. 

Airspeed 
Altitude 
Attitude 
Heading/Course 
Angle  of  Attack 
Flight  Path 
G  Loads 

Evasive/Combat  Maneuvering 
Ground  Operations 

Taxiing,  Braking 


A12  NAVIGATE 

Perform  evaluation  of  the  current  aircraft  status,  position, 
Mission  Plans,  Enroute  Plans,  and  threat  location  and  status.  Pilot 
determination  of  navigation  alternatives  and  operations  and  the 
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execution  of  navigation  procedures. 


Environmental  Effects 
Weather 
Visibility 
Wind 
Icing 

Terrain  Features 
Terrain,  Obstruction  Clearance 
Timing  Constraints 
Fuel  State,  Refueling  Options 
Aircraft  Range 
Target  Location 

Stores  Limitations,  Restrictions 

A13  COMMUNICATE 

Perform  those  actions  that  involve  flight  communications  using 
verbal  or  coded  requests,  responses,  acknowledgments  and 
information  to  other  inflight  aircraft  and  ground  facilities. 


Communications  with  Cell,  AWACS,  ARTCC,  Command  Post  etc. 

Hand/Light/Formation  Signals 
Secure  Voice  Communication  Systems 
Voice  Communication  Systems 

IFF 

JTIDS,  Data  Links 


A14  MANAGE  AIRCRAFT  SYSTEMS 

Perform  those  actions  that  involve  the  control  and  configuration 
of  aircraft  systems  through  monitoring  of  current  status, 
performance,  and  malfunctions. 

Systems  Malfunctions 
Systems  Performance  Levels 
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Aircraft,  Systems  Configuration 
Aircraft  Damage 
Stores  Jettison 


All  FLY  AIRCRAFT 

A111  PERFORM  EVASIVE/COMBAT  MANEUVERS 

Perform  those  actions  that  involve  controlling  aircraft  flight 
path,  direction,  and  energy  state  while  employing  combat  maneuvers 
and  tactics  against  threat  site,  threat  aircraft  and  threat  weapons. 


Desired  Flight  Path  changes 

Changes  to  Aircraft  Attitude,  G  Loads,  Angle  of  Attack, 
Energy  State 
Collision  Avoidance 
Air  Combat  Maneuvers  and  Tactics 

Weapons,  FCS  Capabilities  and  Status 
Target,  Threat  Position  and  Closure  Rates 

All 2  MAINTAIN  DESIRED  HEADING/COURSE 

Perform  those  actions  that  involve  controlling  aircraft  heading. 


All 3  MAINTAIN  DESIRED  ALTITUDE 

Perform  those  actions  that  involve  controlling  aircraft  altitude. 

A114  MAINTAIN  DESIRED  AIRSPEED 

Perform  those  actions  that  involve  controlling  aircraft  airspeed. 


A12  NAVIGATE 
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A121  IDENTIFY  SPATIAL  &  TEMPORAL  DIFFERENCES  BETWEEN 
TARGET  &  OWNSHIP 

Perform  those  actions  that  involve  evaluation  of  target  and 
ownship  geometry  and  closure  rates. 


A122  DETERMINE  PERFORMANCE  OF  AIRCRAFT 

Perform  those  actions  that  involve  the  evaluation  of  ownship 
systems  performance  levels. 

Systems  Performance 
Aircraft  Configuration 
Stores  Limitations,  Effects 

A123  DETERMINE  ENVIRONMENTAL  INFLUENCES  ON  THE 
AIRCRAFT 

Perform  those  actions  that  involve  the  evaluation  of 
environmental  effects  on  ownship  systems  performance. 

Weather 

Heat,  Humidity,  Barometric  Pressure,  Wind,  Icing 
Visibility 

Obstructions  Clearance 
Terrain  Clearance 

A124  MAKE  NAVIGATIONAL  DECISIONS 

Evaluate  aircraft  navigation  status  and  determine  navigation 
actions. 

Own  ship  Position,  Heading,  Altitude,  Airspeed 
Winds 

Desired  Course,  Altitude,  Airspeed 

Threat  Location,  Status 

Time 

Fuel 
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A14  MANAGE  AIRCRAFT  SYSTEMS 


A141  MONITOR  AIRCRAFT  SYSTEMS 

Perform  those  actions  that  involve  the  monitoring  of  aircraft 
systems  and  determination  of  systems  status,  performance,  and 
malfunction  effects. 

A142  CONFIGURE  AIRCRAFT  SYSTEMS 

Perform  those  actions  that  involve  the  configuration  of  ownship 
systems  based  on  systems  performance  levels,  malfunctions  work 
arounds,  task  requirements  and  goals,  and  mission  phase. 

A1421  CONFIGURE  COMM/NAV  SYSTEMS 

Perform  those  actions  that  involve  the  configuration  of 
communications  and  navigation  systems  for  the  current  mission 
phase. 

A1422  CONFIGURE  DEFENSIVE  SYSTEMS 

Perform  those  actions  that  involve  the  configuration  of 
Defensive  systems  for  the  current  mission  phase. 

A1423  CONFIGURE  OFFENSIVE  SYSTEMS 

Perform  those  actions  that  involve  the  configuration  of 
Offensive  systems  for  the  current  mission  phase. 

A1424  CONFIGURE  MISCELLANEOUS  SYSTEMS 

Perform  those  actions  that  involve  the  configuration  of 
configure  Miscellaneous  systems  for  the  current  mission  phase. 
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Flight  Control  Systems  (Autopilot,  Trim,  Stability) 
Propulsion 

Lighting  -  Internal,  External 

Environmental  Control  Systems 

Cabin  Pressure,  Heating,  Cooling,  Oxygen 

Landing  Gear 

Flaps 

Speed  Brakes 
Spoilers 


IEFEND  AIRCRAFT 


A21  MONITOR  THREATS 

Perform  those  actions  that  involve  the  defense  of  the  aircraft 
by  the  monitoring  of  threats  and  the  determination  of  threat  mode, 
status,  and  priority. 


Threat  Mode/Status 
Threat  Priority 
Threat  Location 
Closure  Rates 
Occulting  Status 
Missile  Launch 
Gun  Firing 

Air  Combat  Maneuvering 
Threat  Countermeasure  Decisions 
Threat  Detection,  Identification 
Visual 


Visual  Sighting,  Reflections,  Smoke, 
Tracers,  Missile  Engine  Burn 
Sensors 


Radar,  IR,  EO  Sensors 
RWS/RHAW/TEWS 


Threat  Identification,  Threat  Mode,  Aural 
Warning  Tones,  Prioritization, Symbology 
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Monitor  Threat  Reactions 


A22  COUNTER  SAM/AAA/AI  THREAT 

Perform  those  actions  that  involve  the  defense  of  the  aircraft 
by  implementation  of  SAM/AAA/AI  defensive  systems, 
procedures  and  actions. 

Expendable  Countermeasures  (EXCM) 

Electronic  Countermeasures  (ECM) 

Evasive  Maneuvers 

A23  DETERMINE  THREAT  SUPPRESSION  OPTIONS 

Perform  those  actions  that  involve  the  defense  of  the  aircraft 
by  determination  of  SAM/AAA/AI  threat  suppression  or 
destruction  capability  and  options. 

Threat  Location 
Target  Location 
Occulting 

Threat  Mode,  Status 

Fuel  State 

Time 

Mission  Plan 
Stores 

Course/Route  Constraints,  Options 

A24  COUNTER  CHEMICAL  WEAPON  THREAT 

Perform  those  actions  that  involve  implementation  of 
chemical  defensive  procedures. 


A22  COUNTER  SAM/AAA/AI  THREAT 


181 


A221  COUNTER  SAM/AAA/AI  SYSTEMS 


Pilot  action  in  the  defense  of  the  aircraft  by  employment  of 
defensive  countermeasures  systems  and  procedures  to  counter  the 
threat. 


A222  EXECUTE  SAM/AAA/AI  EVASIVE  MANEUVERS 

Pilot  actions  in  the  defense  of  the  aircraft  by  employment  of 
defensive  evasive  maneuvers  to  counter  the  threat. 


Own  ship  Position,  Velocity 
Threat  Range,  Bearing,  Closure  Rate 
Target  Location 
Occulting 

Threat  Mode,  Status 
Fuel  State 
Stores 

Course/Route  Constraints,  Options 
Interceptor/Missile  Position,  Angle,  Closure  Rate 
Tactics 

A223  COMBINED  SAM/AAA/AI  EFFECTS 

The  combined  result  of  pilot  defensive  actions  against 
SAM/AAA/AI  threats. 


EXCM,  ECM,  Evasive  Maneuvers 


A221  COUNTER  SAM/AAA/AI  SYSTEMS 

A2211  DEPLOY  EXPENDABLE  COUNTERMEASURES 

Pilot  actions  to  dispense  onboard  Expendable  Countermeasures 

Flares 

Chaff 
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Jammers 


A2212  ACTIVATE  ELECTRONIC  COUNTERMEASURES 

Pilot  actions  to  activate  onboard  Electronic  Countermeasures 

Barrage/Noise  Jamming 
Sweep  Jamming 
Deception  Jamming 
Electronic  IR  Jamming 
Spot  Jamming 


1  DEPLOY  EXPENDABLE  COUNTERMEASURES 

A22111  ACTIVATE  CHAFF  PROGRAM 

Pilot  actions  to  activate  the  Chaff  dispensing  system. 

A22112  ACTIVATE  FLARE  PROGRAM 

Pilot  actions  to  activate  the  Flare  dispensing  system. 


A22113  ACTIVATE  EXPENDABLE  JAMMERS  PROGRAM 

Pilot  actions  to  activate  the  Expendable  Jammers  dispensing 
system. 

ACTIVATE  ELECTRONIC  COUNTERMEASURES 

A22121  ACTIVATE  DECEPTION  JAMMING 

Pilot  actions  to  activate  Deception  Jamming  Electronic 
Countermeasures. 
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A22122  ACTIVATE  BARRAGE/NOISE  JAMMING 


Pilot  actions  to  activate  Barrage/Noise  Jamming  Electronic 
Countermeasures. 

A22123  ACTIVATE  SPOT  JAMMING 

Pilot  actions  to  activate  Spot  Jamming  Electronic 
Countermeasures. 

A22124  ACTIVATE  SWEEP  JAMMING 

Pilot  ac'ions  to  activate  Sweep  Jamming  Electronic 
Countermeasures. 

A22125  ACTIVATE  INFRARED  JAMMING 

Pilot  actions  to  activate  Infrared  Jamming  Electronic  Counter 
measures. 


A24  COUNTER  CHEMICAL  WEAPON  THREAT 

A241  DON  CW  GEAR 

Pilot  actions  to  counter  chemical  threats  by  donning,  activating, 
and  checking  available  CW  equipment  (mask,  gloves,  etc.). 

A242  ADMINISTER  PROPHYLACTIC 

Pilot  actions  to  counter  chemical  threats  by  administering  a 
pretreatment  drug. 

A243  ADMINISTER  ANTIDOTE 
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Pilot  actions  to  counter  chemical  threats  by  administering  a 
antidote  treatment  drug. 

A244  EXECUTE  CW  THREAT  EVASION 

Pilot  actions  to  counter  chemical  threats  by  evasive  actions. 

A245  COMBINED  CW  COUNTERMEASURES  EFFECTS 

The  combined  effects  of  pilot  actions  to  counter  chemical 
threats. 

A246  MONITOR  CW  THREAT  SITUATION 

Pilot  actions  to  monitor  chemical  threats  and  determine 
subsequent  actions. 


PERFORM  OFFENSIVE  ACTIONS 

A31  PERFORM  OFFENSIVE  AIR-AIR 

Perform  those  actions  that  involve  offensive  actions  against 
airborne  threats. 

A32  PERFORM  OFFENSIVE  AIR-GROUND 

Perform  those  actions  that  involve  offensive  actions  against 
ground  targets. 


A31  PERFORM  OFFENSIVE  AIR-AIR 
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A311  CONFIGURE  AIRCRAFT  AIR-AIR 

Perform  those  actions  that  involve  the  activation  and 
configuration  tasks  required  for  an  offensive  air-air  engagement. 

A312  ATTACK  AIR-AIR  TARGET 

Perform  those  actions  that  involve  air  combat  maneuvering, 
target  acquisition,  and  weapons  delivery  against  an  air-air  target. 

A313  DISENGAGE  AIR-AIR  TARGET 

Perform  those  actions  that  involve  disengagement  from  air 
combat  against  an  air-air  target. 


A311  CONFIGURE  AIRCRAFT  AIR-AIR 

A3111  CONFIGURE  FIRE  CONTROL  &  WEAPON  SYSTEMS 

Perform  those  actions  that  involve  the  configuration  of  the  Fire 
Control  System  and  Weapon  Systems 

BIT  Checks 

Select  Air-Air  Master  Mode 
Weapons  Selection 

Missile  Selection,  Station  Selection 
Arming 

Tuning  -  cooling,  battery  power,  guidance, 
seekers 

Guns  Select/Arming 


A3112  CONFIGURE  RADAR  SYSTEM 
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Perform  those  actions  that  involve  the  configuration  of  the 
Radar  System. 

Channel 
Radar  Mode 
Elevation  Strobe 
Brightness 
Sector  Coverage 
Polarity 
Contrast 
Symbology 

A3113  CONFIGURE  COUNTERMEASURE  SYSTEMS 

Perform  those  actions  that  involve  the  configuration  of  the 
Countermeasures  Systems. 

ECM  Tuning,  Programs 
Flare  Program 

Burst  Count,  Interval 
Chaff  Program 

Salvo  Count,  Interval 
Burst  Count,  Interval 
Radar  Warning  Equipment 
Display  Controls 
Aural  Tones  Volume 


A3114  CONFIGURE  MISCELLANEOUS  SYSTEMS 

Perform  those  actions  that  involve  the  configuration  of 
miscellaneous  aircraft  systems. 


Configure  Lighting 

Internal  Lighting 
External  Lighting 
Configure  Fuel 
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Jettison  Tanks 
Transfer  Fuel 
Check  Fuel  Status 

Configure  Electro-Optics  Search/Targeting  System 
Configure  HUD 


A312  ATTACK  AIR-AIR  TARGET 

A3121  ACQUIRE  AIR-AIR  TARGET 

Perform  those  actions  that  involve  the  acquisition  and 
designation  of  the  target  for  weapons  release. 

Defensive  Maneuvering 
Offensive  Maneuvering 
Threat  Status 
Threat  Location 
Own  ship  Status 
Sensor  Status,  Capabilities 

A3222  RELEASE  WEAPONS 

Perform  those  actions  that  involve  the  delivery  of  weapons 
against  an  airborne  target. 

A3122  EXECUTE  AIR  COMBAT  MANEUVERS  &  TACTICS 

Perform  those  actions  that  involve  aircraft  maneuvering  against 
an  airborne  target.  Defensive  Maneuvering 

Threat  Actions 

Attack  Geometry 

Own  ship  Status 

Own  ship/Threat  Energy  States 

Rules  of  Engagement 

Air-Air  Combat  Tactics 
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Weapon  Delivery  Parameters 

FCS  and  Weapon  System  Capabilities,  Limitations 

A3123  RELEASE  AIR-AIR  WEAPONS 

Perform  those  actions  that  involve  the  delivery  of  weapons 
against  the  airborne  target. 


A32  PERFORM  OFFENSIVE  AIR-GROUND 

A321  INGRESS  TO  IP 

Perform  those  actions  that  involve  a  low  level  ingress  to  the 
target  IP.  Completion  of  all  pre-attack  phase  tasks  and  aircraft 
configuration  in  order  to  minimize  all  required  actions  during  the 
attack  phase. 

Emphasis:  Limit  Attack  Concerns  Prior  to  IP 

Lessen/Relieve  Workload  During  Ingress  to  IP 

Check  Aircraft  Systems  Status 
Weapons,  FCS  Status 
Fuel  Status 
Perform  BIT  Checks 


Abort  Attack  Decisions 

Systems  Malfunctions 
Timinq  Problems 
Fuel 

Unanticipated  Threats 
Threat  Actions 
Support  Problems 


Radar/IR  Check  for  Bandits 
Cell/AWACS  coordination 
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A322  ATTACK  TARGET 


Perform  those  actions  that  involve  the  attack  target  segment  of 
the  mission  -  including  target  acquisition  and  weapon  release. 

Hack  Clock 

Fly  an  Unpredictable  Flight  Path 
Approach  Terrain 
-  Target  Elevation 
Threat  Locations 
Low  Altitude  Ingress 
Weather  Maneuvering 
Obstruction/Terrain  Maneuvering 

Time-Over-Target  Monitoring  for  attack  airspeed  control 
Minimum  Altitude,  Maximum  Speed 
Lowers  Accuracy 
Increases  Survivability 
Lowers  Detection, 

Decreases  Acquisition  Time 
Critical  Weapons  Factors 
Fuzzing  Time 
Release  Altitude 
Frag  Damage 
Impact  Angles 
Seeker  Limits 
Stores  Airspeed  Limits 
Stores  G  Limits 

Weapon/Seeker  Cooling  Requirements 
Weapon  Battery  Limits 
Weapon  Tuning 
Fuel  Usage  vs.  Speed 

CCIP  Attack 

Pop-up  Not  as  Essential  for  Weapons  Delivery 
Since  Only  a  Visual  Acquisition  Required 

CCRP  Attack 
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Computer  Driven,  Requires  Target  Designation 
on  HUD,  Radar,  INS 


Acquisition  Maneuvers 
Bombing  Mode 
Loft 

Dive  Toss 
Level  Bombing 

Pop-Up 

Pull-Down 

Off-Axis  (Angle-Off)  Maneuver 
Faster  Acquisition 
Unpredictable  Flight  Path 
More  Maneuvering  Required  Less  Tracking  Time 
Possible 

On-axis  Maneuver 

Acquisition  Parameters 
Altitude 

Line  of  Sight  to  Target 

Environmental  Factors 
Ambient  Light 
Smoke 
Haze/Fog 

Blowing  Snow/Sand 
Rain 

Sun/Moon  Angle 
Visibility 

Environmental  Acquisition  Problems 
Increases  Exposure  Time 
Decreases  Target  Acquisition  Time 
Formation  Spacing 
Shortens  Release  Range 
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Visual  Limitations/Restrictions 
Range  to  Target 
Altitude 
Airspeed 

Aspect  -  Azimuth,  Elevation 

Winds 

Decreased  Accuracy 
Blowing  Obscurations 

Ground  Beacons,  Smoke  Markers,  FAC  Directions 

Expected  Range,  Bearing  to  Target  vs.  Actual 

Target  Features 

Surrounding  Terrain 
Features 
Type 
Colors 

Size 

Features 

Spacing,  Orientation 
Color 

Reflections 
Contrast  Ratio 

Target  Electronic  Emissions 

Mobile  Targets  -  Visual  Designation  Desired 

Intelligence,  Target  Information 

Target  Designator  Box  Cues 

Closure  Rates,  Time  for  Acquisition 

Target  Designation 

Visual  -  Eyes,  HUD 
EO,  TV,  FLIR 
Radar 
INS 

Dead  Reckoning 
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Damage  Assessment  -  Pre-Attack 
Evaluate  target  damage  level 

A323  DISENGAGE  TARGET 

Perform  those  actions  that  involve  the  evaluation  of  target  and 
attack  status  for  determination  of  attack  pass  abort,  re-attack  or 
tc  rmination  options. 

A324  EGRESS  TARGET  AREA 

Perform  those  actions  that  involve  the  Post  attack  Evaluation  of 
target  and  attack  status  for  determination  of  re-attack  or 
termination  and  RTB  options. 


A321  INGRESS  TO  IP 

A3211  FLY  TO  IP 

Perform  those  functions  performed  during  the  ingress  to  the  IP. 
Tasks  include  navigation,  cel!  communications  and  coordination, 
attaining  proper  timing,  threat  monitoring. 

A3212  CONFIGURE  AIRCRAFT 

Perform  those  actions  that  involve  Pilot  conducting  aircraft 
operations  in  terms  of  configuring  the  various  onboard  aircraft 
systems. 

A3213  PERFORM  NAV  UPDATE 

Perform  those  actions  that  are  necessary  to  identify  the  IP  and 
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perform  an  INS/Navigation  Update. 


A3211  FLY  TO  IP 

A32111  PERFORM  LOW  LEVEL  PENETRATION 

Perform  those  actions  that  involve  the  performance  of  a  low 
level  target  penetration. 

A32112  COORDINATE  WITH  CELL/ AW  ACS 

Perform  those  actions  that  involve  the  coordination  of  the 
ownship  with  the  cell/AWACS/Command  members. 

A32113  SOLVE  TIMING  PROBLEMS 

Perform  those  actions  that  involve  meeting  the  time 
requirements  for  navigation,  weapon  delivery,  and  mission 
coordination. 

A32114  CHECK  TARGET  AREA  FOR  BANDITS 

Perform  those  actions  that  involve  searching  the  target  area  and 
approach  for  enemy  aircraft. 


A3212  CONFIGURE  AIRCRAFT 

A32121  CONFIGURE  FIRE  CONTROL  &  WEAPON  SYSTEMS 

Perform  those  actions  that  involve  the  configuration  of  the  Fire 
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Control  System  and  Weapon  Systems 


Perform  BIT  Checks 

Select  Air-Ground  Master  Mode 

Weapons  Selection 

Select  Weapon  Delivery  Mode 
Weapon  Selection,  Station  Selection 
Fuzzing 

Nose/Tail 

Type  Fuzzing  -  Proximity,  Time,  Impact 
Arming 
Interval 

Release  Parameters 
Pull-up  Timers 
Release  Timers 

HUD  Reticle  Depression 

Guns  Selection,  Firing  Rate 

Weapons  Arming/Tuning 
Cooling 
Battery  Power 
Guidance 
Seekers 


A32122  CONFIGURE  RADAR  SYSTEM 

Perform  those  actions  that  involve  the  configuration  of  the 
Radar  System. 


Channel 
Radar  Mode 
Elevation  Strobe 
Brightness 
Sector  Coverage 
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Contrast 

Symbology 

A32123  CONFIGURE  COMMUNICATION  SYSTEM 

Perform  those  actions  that  involve  the  configuration  of  the 
Communication  System. 


Secure  Voice 

Silent  Communications 

IFF 

JTIDS 


A32124  CONFIGURE  NAVIGATION  SYSTEM 

Perform  those  actions  that  involve  the  configuration  of  the 
Navigation  System. 

Tune  Radios 

Select  Navigation  Mode 

Waypoint  Selection 


A32125  CONFIGURE  COUNTERMEASURE  SYSTEMS 

Perform  those  actions  that  involve  the  configuration  of  the 
Countermeasures  Systems. 


ECM  Tuning,  Programs 

Flare  Program 

Burst  Count,  Interval 

Chaff  Program 

Salvo  Count,  Interval 
Burst  Count,  Interval 


196 


Radar  Warning  Equipment 
Display  Controls 
Aural  Tones  Volume 

A32126  CONFIGURE  MISCELLANEOUS  SYSTEMS 

Perform  those  actions  that  involve  the  configuration  of 
miscellaneous  aircraft  systems. 

Configure  Lighting 

Internal  Lighting 
External  Lighting 

Configure  Fuel 

Jettison  Tanks 
Transfer  Fuel 
Check  Fuel  Status 

Configure  Electro-Optics  Search/Targeting  Systems 

Configure  HUD 

Display  Symbology 

Brightness 

Camera 


A32121  CONFIGURE  FIRE  CONTROL  &  WEAPON  SYSTEMS 

A321211  SELECT  A/G  MASTER  MODE 

Pilot  actions  to  select  the  A/G  Master  Mode. 

A321212  SELECT  WEAPON  DELIVERY  MODE 

Pilot  actions  to  select  the  desired  Weapon  Delivery  Mode. 

Selected  Bombing  Mode 

A321213  SELECT  WEAPON  PARAMETERS 
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Pilot  actions  to  select  weapon  parameters. 

Weapon  Type  Selection 
Station  Selection 
Fuzzing  Parameters 

A321214  SELECT  WEAPON  RELEASE  PARAMETERS 

Pilot  actions  to  select  the  weapon  release  parameters. 

Interval 

Timers 

Adjust  HUD  Reticule  Depression 

A321215  ARM  WEAPONS 

Pilot  actions  to  tune,  arm  selected  weapons. 

A/G  Weapons  Tuned,  Armed 
Gun  Arming,  Firing  Rate 

A321216  PERFORM  FCS  &  WEAPONS  BIT  CHECKS 

Pilot  actions  to  execute  Built-in-Test  diagnostics  for  the  Fire 
Control  System  and  Weapons  System. 


A32122  CONFIGURE  RADAR  SYSTEM 

A321221  PERFORM  RADAR  BIT  CHECKS 

Pilot  actions  to  execute  Built-In-Test  diagnostics  for  the  Radar 
System. 

A321222  SELECT  RADAR  CONTROL  PANEL  FUNCTIONS 

Pilot  selection  of  Radar  Control  Panel  Functions. 


A321223  ADJUST  RADAR  DISPLAY 

Pilot  adjustment  of  Radar  Display  Controls. 
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A321224  ADJUST  ANTENNA  ELEVATION 

Pilot  adjustment  of  Radar  Antenna  Elevation  Angle 
Controls. 

Radar  beam  grazing  angle 
Radar  energy  returns 


A321222  SELECT  RADAR  CONTROL  PANEL  FUNCTIONS 


A3212221  SELECT  RADAR  MODE 

Pilot  selection  of  Radar  Mode  parameters. 

Ground  Map  Modes 
A/A  Modes 
NAV  Modes 


A3212222  SELECT  SEARCH  PARAMETERS 

Pilot  selection  of  Radar  Search  parameters. 


Range 

Azimuth  Scan  Width 
Number  of  Elevation  Bars 

A3212223  SELECT  FREQUENCY  PARAMETERS 

Pilot  selection  of  Radar  Frequency  parameters. 


RF  Channel 
PRF 
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A3212224  SELECT  MISCELLANEOUS  PARAMETERS 

Pilot  selection  of  Miscellaneous  Radar  Control  functions. 


Target  History 
Marker  Intensity 
Map  Freeze 


A321223  ADJUST  RADAR  DISPLAY 

A3212231  ADJUST  SYMBOLOGY  INTENSITY 

Pilot  adjustment  of  Radar  Display  Symbology  Controls. 

A3212232  ADJUST  VIDEO  GAIN 

Pilot  adjustment  of  Radar  Display  Video  Gain  Controls. 

A3212233  ADJUST  VIDEO  INTENSITY 

Pilot  adjustment  of  Radar  Display  Video  Intensity  Controls. 

A3212234  ADJUST  VIDEO  CONTRAST 

Pilot  adjustment  of  Radar  Display  Video  Contrast  Controls. 

A32123  CONFIGURE  COMMUNICATION  SYSTEM 

A321231  SELECT  COMMUNICATION  MODE 

Pilot  selection  of  Communication  Mode  parameters. 

A321232  TUNE  COMMUNICATION  RADIOS 

Pilot  selection  of  Communication  Radio  frequency/channel. 
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Selected  Channels/Frequencies 


A321233  SELECT  SECURE  VOICE  MODES 

Pilot  selection  of  Secure  Voice  Communication  Modes. 

A321234  SELECT  IFF  MODES 

Pilot  selection  of  IFF  Mode  &  Interrogation  parameters. 

IFF  Mode 

IFF  Transponder  Codes 


A32124  CONFIGURE  NAVIGATION  SYSTEM 

A321241  PERFORM  INS  BIT  CHECKS 

Pilot  actions  to  execute  Built-in-Test  diagnostics  for  INS 
System. 

A321242  SELECT  NAVIGATION  MODES 

Pilot  selection  of  Navigation  Mode  parameters. 


A321243  VERIFY  INS  COORDINATES 

Pilot  actions  to  check  actual  coordinates  vs.  INS  coordinates. 

Waypoint  Location  Coordinate  Entries 

Own  ship  Actual  Position 

INS  Computed  Own  ship  Position 

A321244  TUNE  NAVIGATION  RADIOS 

Pilot  selection  of  Navigation  radio  frequencies/channels. 


A32125  CONFIGURE  COUNTERMEASURES  SYSTEMS 

A321251  CONFIGURE  ELECTRONIC  COUNTERMEASURES  SYSTEM 

Perform  those  actions  that  involve  the  configuration  of  the 
Electronic  Countermeasures  System. 


A321252  CONFIGURE  EXCM  SYSTEM 

Perform  those  actions  that  involve  the  configuration  of  the 
Expendable  Countermeasures  System. 


A321252  CONFIGURE  EXCM  SYSTEM 

A3212521  SELECT  FLARE  PARAMETERS 

Pilot  selection  of  Flare  Program  parameters. 

Burst  Count,  Interval 

A3212522  SELECT  CHAFF  PARAMETERS 

Pilot  selection  of  Chaff  Program  parameters. 

Burst  Count,  Interval 
Salvo  Count,  Interval 

A3212523  SELECT  EXCM  PROGRAM 

Pilot  selection  of  Expendable  Countermeasures  Program. 

A3212524  ARM  EXCM  SYSTEM 
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Pilot  Arming  of  the  Expendable  Countermeasures  System. 


A32126  CONFIGURE  MISC.  SYSTEMS 

A321261  CONFIGURE  FUEL  SYSTEM 

Perform  those  actions  that  involve  the  configuration  of  the  Fuel 
System. 

Jettison  Tanks 
Transfer  Fuel 
Check  Fuel  Status 

A321262  ASSIGN  TARGET  DESIGNATOR  CONTROL  (TDC) 

Pilot  assignment  of  the  TDC  for  designation  from  the  desired 
display. 

HUD,  Radar,  HSI 

A321263  ADJUST  LIGHTING 

Pilot  adjustment  of  aircraft  lighting  controls. 


Interior,  Exterior 

A321264  ADJUST  ELECTRONIC  DISPLAYS 

Pilot  adjustment  of  various  electronic  display  controls. 


Weapons  Video 

Multi-function  Displays 

Electro-Optical  Search/Acquisition  Systems 

A321265  ADJUST  HUD  CONTROLS 

Pilot  adjustment  of  the  Heads  Up  Display  controls. 
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Symbology 
Brightness 
Camera  Controls 


A322  ATTACK  TARGET 

A3221  ACQUIRE  TARGET 

Perform  those  actions  that  involve  the  visual,  radar,  or  electro- 
optical  acquisition  and  designation  of  the  target  for  weapons 
release. 


Start  Countermeasures 

Evasive  Maneuvers/Jinking 

Preventative  Countermeasures  (EXCM,  ECM) 

Threat  Status 
Ignore 

Counter  Measures 
Evade  Defenses 
Abort  Attack  Pass 
Abort  Attack 
Suppress/Destroy  Threat 

Considerations 

Lower,  Slower  -  More  accurate,  riskier 

A3222  RELEASE  WEAPONS 

Perform  those  actions  that  involve  the  delivery  of  weapons 
against  the  ground  target. 


A3223  ASSESS  TARGET  DAMAGE 

Determine  target  damage  levels  and  degree  of  success  with 
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respect  to  acceptable  levels  of  damage. 


A3221  ACQUIRE  TARGET 

A32211  EXECUTE  ACQUISITION  MANEUVER 

Perform  those  actions  that  involve  maneuvering  the  aircraft  so 
that  target  acquisition  can  be  accomplished  and  maintained. 

Threat  Exposure 

Threat  Location 
Threat  Mode,  Status 
Occulting,  Line  of  Sight 

Weapon  Release  Parameters 
Weapon  Delivery  Mode 
Weapon  Fuzzing 
Ground/Obstacle  Clearance 

Pop-Up  Angle 
Pop-Up  G's 
Pop-Up  Altitude 
Pull-Down  G's 

A32212  PERFORM  TARGET  ACQUISITION 

Perform  those  actions  that  involve  visual  or  sensor  target 
acquisition. 


A32213  DESIGNATE  TARGET 

Perform  those  actions  that  involve  weapon  system  or  sensor 
target  designation  for  weapons  delivery  solutions. 
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A32212  PERFORM  TARGET  ACQUISITION 

A322121  PERFORM  VISUAL  ACQUISITION 

Perform  those  actions  that  involve  visual  acquisition  of  the 
target  in  conjunction  with  HUD  and  electro-optic  systems. 

A322122  PERFORM  RADAR  ACQUISITION 

Perform  those  actions  that  involve  radar  acquisition  of  the 
target. 


A322123  PERFORM  INS  ACQUISITION 

Perform  those  actions  that  involve  INS  acquisition  of  the  target. 


A322124  PERFORM  DEAD  RECKONING  ACQUISITION 

Perform  those  actions  that  involve  dead  reckoning  navigation  to 
the  target. 

Timing 

Airspeed 

Position 

Distance 

Heading 

Timing 

Coordinated  Attacks 
Frag  Avoidance 
Secondary  Explosions 


A32222  TRACK  TARGET 

Perform  those  actions  that  involve  maintaining  a  flight  path 
that  will  allow  accurate  weapon  delivery.  In  addition,  assess  current 
target  damage  for  attack  abort  decisions. 


206 


Attack  Run  Target  Damage  Assessment 
Continue  Attack 

Abort  Attack  (Acceptable  Damage  Levels) 
Track  Target 

Release  Airspeed 
Dive  Angle 
G's 

Release  Altitude 
Winds 

Azimuth  Tracking/Ground  Path 
Fly  to  Release  Point,  Roll-in  Point 


A32223  FLY  TO  RELEASE  POINT 

Perform  those  actions  that  involve  maintaining  a  flight  path 
that  will  allow  accurate  weapon  delivery  at  the  necessary  weapon 
release  point. 

Track  Target  to  Release  Point 


A32224  RELEASE  WEAPONS 

Perform  those  actions  that  involve  munitions  delivery. 

Pickle 

Trigger 

Weapon  Delivery 

Pop-up  Well  Before  Minimum  Attack  Perimeter  (MAP) 
MAP  Based  on  Roll-out,  Target  Tracking,  Pickle 

TD  Box  Location 

CCIP  Display  and  Target  Location 
CCRP  Steering  Cues 
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Velocity  Vector,  Attitude,  Altitude,  Airspeed, 

Time  to  Go,  G’s,  Release  Cue,  Pull-up  Cue 

AOA,  HSI,  ADI,  INS,  HUD/Optical  Sight,  Radar,  Au^io 
Tones,  Release/Pull-up  Cues,  Pull-up  Timers 

Drag  Options 
Dive  Angle 
Release  Speed 

Target  Tracking  -  MAP  -  Release  Point  Coordination 

Timing/Spacing 

Frag  Damage 
Cell  Coordination 


A323  DISENGAGE  TARGET 

A3231  EXECUTE  ESCAPE  MANEUVER 


Perform  those  actions  that  involve  aircraft  maneuvers  to  clear 
the  target  area  and  avoid  threats. 

Escape  Maneuvers 

Terrain  Avoidance,  Max  Airspeed 
Visibility 
Ceilings 
Fuel  Usage 
High  G  Pull-up 
High  G  Level  Turn 
High  G  Pull-up,  Turn 

Fragmentation,  Secondary  Explosions  Avoidance 
Cell  Coordination 
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A3232  RE-JOIN  CELL 


Perform  those  actions  that  involve  reforming  with  the  attack 
package/cell  members. 


A3233  EVALUATE  AIRCRAFT  STATUS 

Perform  those  actions  that  involve  determination  of  post-attack 
aircraft  systems,  aircraft  status,  and  pilot  physiological  condition 
and  capabilities. 

BIT  Checks 
Visual  Inspection 
Safe  Weapons 

Safe  Countermeasures  Systems 

A3234  DETERMINE  RE-ATTACK  PLANS 

Perform  those  actions  that  involve  determination  of  re-attack 
options  based  on  attack  status,  aircraft  status,  and  target  status. 


Target  Damage  Assessment  -  Post  Attack 
Acceptable  Levels  of  Damage 
Unsuccessful  Release 

Execute  Countermeasures 
Preventative  Actions 

Follow-Up  Actions 
Abort  Mission 
Reattack  Target 
Secondary/Alternate  Target 


A3241  DETERMINE  MISSION  OPTIONS 


Perform  those  actions  that  involve  determination  of  re-attack 
options,  abort  decisions,  alternate  target  selection  and  attack  and 
mission  disengagement. 

Determine  Actions/Options 
Available  Time 
Available  Stores 
Aircraft  Systems  Status 
Battle  Damage 
Pilot/Crew  Condition 

Fatigue,  Wounds,  Chemical  Exposure 
Downed  Aircraft 
Alternate  Target  Decisions 

A3242  RE-FUEL  AIRCRAFT 

Perform  those  actions  that  involve  inflight  aircraft  refueling. 

Alternate  Target  Decisions 
Mission  Plan 
Fuel  State 

A3243  PROCEED  TO  ALTERNATE/SECONDARY  TARGET 

Perform  those  actions  that  involve  navigation  to  alternate  or 
secondary  targets. 

Available  Fuel,  Refueling  Options 
Alternate  Along  Egress  Route 

A3244  RETURN  TO  BASE 

Perform  those  actions  that  involve  egress  to  recovery  base. 
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A3245  CONFIGURE  AIRCRAFT  FOR  EGRESS 

Perform  those  actions  that  involve  the  configuration  of  the 
aircraft  for  egress  to  the  recovery  base. 


Configure  Countermeasures  Systems 
Configure  Comm/Nav  Systems 
Configure  Radar  Systems 
Configure  Misc.  Systems 
Configure  FCS/Weapon  Systems 

Air-Air  Weapon  Selection,  Arming 
Jettison  Stores 

A3246  COORDINATE  WITH  CELL/AWACS 

Perform  those  actions  that  involve  Cell/AWACS  coordination 
and  communication. 


A4  MANAGE  MISSION  PHASE 

A41  EXECUTE  MISSION  PHASES 

Perform  those  actions  that  involve  completion  of  the  current 
mission  phase  objectives  and  making  enroute  plans  and  revisions. 

A42  ASSESS  MISSION  PHASES 

Perform  those  decisions  that  involve  the  completion,  transition, 
and  continuation  of  the  current  mission  phase  based  on  Mission 
Status  and  conditions. 


A41  EXECUTE  MISSION  PHASES 
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A411  EXECUTE  TAKEOFF 


Perform  those  actions  that  involve  takeoff  phase  tasks. 

A412  EXECUTE  OUTBOUND/INBOUND  CRUISE 

Perform  those  actions  that  involve  cruise  phase  tasks. 

A413  EXECUTE  INGRESS/EGRESS 

Perform  those  actions  that  involve  ingress/egress  phase  tasks. 

A414  EXECUTE  TARGET  ENGAGEMENT 

Perform  those  actions  that  involve  target  attack  phase  tasks. 

A415  EXECUTE  RECOVERY  AND  LANDING 

Perform  those  actions  that  involve  aircraft  recovery  and  landing 
tasks. 


A42  ASSESS  MISSION  PHASES 

A421  ASSESS  SYSTEM 

Determine  system  capabilities  and  status  based  on  aircraft 
performance,  battle  damage  and  malfunctions. 

A422  ASSESS  ENVIRONMENT 

Determine  environmental  effects  on  aircraft  capabilities  and 
mission  performance. 

A423  ASSESS  PHASE  OBJECTIVE 
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Determine  the  status  of  current  phase  objectives,  mission 
status,  capabilities,  and  current  actions. 


A424  DEFINE  COURSE  OF  ACTION 

Determine  subsequent  plans  and  actions  necessary  to  achieve 
mission/phase  objectives  based  on  current  status,  conditions  and 
possible  alternatives. 

Mission  Abort/Continuance  Decisions 
Malfunctions  Effects 
Battle  Damage 

Pilot  Physiological  Condition 
Ejection  Decisions 
Downed  Aircraft 
Fuel  State 
T  raffic 

Weather,  Environmental  Effects 
Threat  Mode,  Status 
Malfunction  Effects 
Systems  Status 
Available  Stores 
Ownship  Position 
Mission  Plan 

Mission  Goals 
Abort  Criteria 

Doctrine/Rules  of  Engagement 
Attack  Package/Cell  Status 
Target  Location,  Status 
Attack  Status 
Time 
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APPENDIX  D 


STORYBOARD  PROTOTYPE 
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Storyboard  Glossary  of  Design  Object 
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